diff options
| author | Kirill Rysin <[email protected]> | 2025-09-28 04:32:57 +0200 |
|---|---|---|
| committer | GitHub <[email protected]> | 2025-09-28 05:32:57 +0300 |
| commit | db154818a7ba839a7dc7b99f28fa5baec43e00ec (patch) | |
| tree | 12ec3296da729b71cb583bd038537d3c72458af2 | |
| parent | ce007905a0d0675a73d4aee3d5f7c5963923f1c0 (diff) | |
Mute bot: statistic in telegram (#25846)
| -rw-r--r-- | .github/scripts/telegram/README.md | 10 | ||||
| -rw-r--r-- | .github/scripts/telegram/README_parse_and_send.md | 261 | ||||
| -rw-r--r-- | .github/scripts/telegram/README_team_issues.md | 9 | ||||
| -rw-r--r-- | .github/scripts/telegram/README_telegram.md | 9 | ||||
| -rw-r--r-- | .github/scripts/telegram/parse_and_send_team_issues.py | 983 | ||||
| -rw-r--r-- | .github/scripts/telegram/send_telegram_message.py | 250 | ||||
| -rw-r--r-- | .github/workflows/create_issues_for_muted_tests.yml | 4 | ||||
| -rw-r--r-- | .github/workflows/weekly_telegram_update.yml | 46 |
8 files changed, 1514 insertions, 58 deletions
diff --git a/.github/scripts/telegram/README.md b/.github/scripts/telegram/README.md index eec4a5c15bd..8e376e1a166 100644 --- a/.github/scripts/telegram/README.md +++ b/.github/scripts/telegram/README.md @@ -17,17 +17,19 @@ General-purpose script for sending messages or file contents to Telegram channel **Documentation:** [README_telegram.md](README_telegram.md) ### đĨ `parse_and_send_team_issues.py` -Specialized script for parsing GitHub issues and sending team-specific notifications. +Specialized script for parsing GitHub issues and sending team-specific notifications. Supports two modes: immediate updates and periodic trend reports. **Features:** -- Parse team issues from formatted results -- Send separate messages for each team +- **On-Mute-Change Updates**: Parse team issues from formatted results and send immediate notifications +- **Periodic Trend Updates**: Send weekly/monthly trend reports with statistics and charts - Support for multiple responsible users per team - Team lead mentions in messages - Support for message threads - Dry run mode for testing +- Automatic team discovery from YDB data +- Fallback to default channel for teams without specific configuration -**Documentation:** [README_team_issues.md](README_team_issues.md) +**Documentation:** [README_parse_and_send.md](README_parse_and_send.md) ## Configuration diff --git a/.github/scripts/telegram/README_parse_and_send.md b/.github/scripts/telegram/README_parse_and_send.md new file mode 100644 index 00000000000..3a9b6981519 --- /dev/null +++ b/.github/scripts/telegram/README_parse_and_send.md @@ -0,0 +1,261 @@ +# Parse and Send Team Issues + +Script to parse GitHub issues results and send separate messages for each team with muted tests statistics. Supports two modes: immediate updates and periodic trend reports. + +## Quick Start + +### Mode 1: On-Mute-Change Updates (Default) +Send immediate notifications about new muted tests: + +```bash +# Basic usage with YDB statistics +python .github/scripts/telegram/parse_and_send_team_issues.py \ + --on-mute-change-update \ + --file "formatted_results.txt" \ + --team-channels '{"teams": {"team-name": {"responsible": ["@user1", "@user2"], "channel": "channel-name"}}, "channels": {"channel-name": "123456789/1"}}' \ + --bot-token "YOUR_BOT_TOKEN" + +# With custom YDB settings +python .github/scripts/telegram/parse_and_send_team_issues.py \ + --on-mute-change-update \ + --file "formatted_results.txt" \ + --team-channels '{"teams": {"team-name": {"responsible": ["@user1"], "channel": "channel-name"}}, "channels": {"channel-name": "123456789/1"}}' \ + --bot-token "YOUR_BOT_TOKEN" \ + --ydb-endpoint "grpcs://ydb.serverless.yandexcloud.net:2135" \ + --ydb-database "/ru-central1/b1g8ejbrie0sfh5k0j2j/etn8l4e3hbti8k4n5g2e" \ + --ydb-credentials "path/to/credentials.json" + +# Dry run (show messages without sending) +python .github/scripts/telegram/parse_and_send_team_issues.py \ + --on-mute-change-update \ + --file "formatted_results.txt" \ + --team-channels '{"teams": {"team-name": {"responsible": ["@user1"], "channel": "channel-name"}}, "channels": {"channel-name": "123456789/1"}}' \ + --dry-run + +# Skip statistics fetch +python .github/scripts/telegram/parse_and_send_team_issues.py \ + --on-mute-change-update \ + --file "formatted_results.txt" \ + --team-channels '{"teams": {"team-name": {"responsible": ["@user1"], "channel": "channel-name"}}, "channels": {"channel-name": "123456789/1"}}' \ + --bot-token "YOUR_BOT_TOKEN" \ + --no-stats +``` + +### Mode 2: Periodic Trend Updates +Send weekly or monthly trend reports with statistics and charts: + +```bash +# Weekly trend updates +python .github/scripts/telegram/parse_and_send_team_issues.py \ + --period-update week \ + --team-channels '{"default_channel": "main_channel", "teams": {"team-name": {"responsible": ["@user1"], "channel": "main_channel"}}, "channels": {"main_channel": "123456789/1"}}' \ + --bot-token "YOUR_BOT_TOKEN" \ + --ydb-endpoint "grpcs://ydb.serverless.yandexcloud.net:2135" \ + --ydb-database "/ru-central1/b1g8ejbrie0sfh5k0j2j/etn8l4e3hbti8k4n5g2e" + +# Monthly trend updates with debug plots +python .github/scripts/telegram/parse_and_send_team_issues.py \ + --period-update month \ + --team-channels '{"default_channel": "main_channel", "teams": {"team-name": {"responsible": ["@user1"], "channel": "main_channel"}}, "channels": {"main_channel": "123456789/1"}}' \ + --bot-token "YOUR_BOT_TOKEN" \ + --ydb-endpoint "grpcs://ydb.serverless.yandexcloud.net:2135" \ + --ydb-database "/ru-central1/b1g8ejbrie0sfh5k0j2j/etn8l4e3hbti8k4n5g2e" \ + --debug-plots-dir "/path/to/debug/plots" \ + --dry-run +``` + +## Parameters + +### Mode Selection (Required - Choose One) +- `--on-mute-change-update` - Default mode: send updates about new muted tests (requires --file) +- `--period-update {week,month}` - Send periodic trend updates (no --file required) + +### Required +- `--team-channels` - JSON string mapping teams to their channel configurations (or use TEAM_CHANNELS env var) +- `--file` - Path to file with formatted results (required only for --on-mute-change-update mode) + +### Optional +- `--bot-token` - Telegram bot token (or use TELEGRAM_BOT_TOKEN env var) +- `--delay` - Delay between messages in seconds (default: 2) +- `--dry-run` - Parse and show teams without sending messages +- `--test-connection` - Test Telegram connection only +- `--message-thread-id` - Thread ID for group messages (optional) +- `--max-retries` - Maximum number of retry attempts for failed messages (default: 5) +- `--retry-delay` - Delay between retry attempts in seconds (default: 10) + +### YDB Statistics +- `--ydb-endpoint` - YDB database endpoint (or use YDB_ENDPOINT env var) +- `--ydb-database` - YDB database path (or use YDB_DATABASE env var) +- `--ydb-credentials` - Path to YDB service account credentials JSON file (or use YDB_SERVICE_ACCOUNT_KEY_FILE_CREDENTIALS env var) +- `--no-stats` - Skip fetching muted tests statistics from YDB (only for --on-mute-change-update) +- `--include-plots` - Include trend plots in messages (requires matplotlib) +- `--debug-plots-dir` - Directory to save debug plot files (enables debug mode) +- `--use-yesterday` - Use yesterday's data for development convenience + +## Features + +### On-Mute-Change Updates Mode +- â
Automatic parsing of team issues from formatted results +- â
Separate messages for each team +- â
Muted tests statistics from YDB (total, today, and minus today counts) +- â
Monthly trend plots with matplotlib +- â
Team-specific channel routing +- â
Responsible user mentions +- â
Markdown formatting support +- â
Delay between messages to avoid API limits +- â
Message thread support +- â
Automatic retry mechanism +- â
Dry run mode for testing +- â
Connection testing + +### Periodic Trend Updates Mode +- â
Weekly and monthly trend reports +- â
Automatic team discovery from YDB data +- â
Team blacklist support (exclude specific teams from periodic updates) +- â
Fallback to default channel for teams without specific configuration +- â
Trend charts with 30-day history +- â
Period-over-period change calculations +- â
Color-coded statistics (red for increases, green for decreases) +- â
Dashboard links for each team +- â
Responsible user mentions (if team configured) +- â
Team hashtags for easy filtering + +## Message Format + +### On-Mute-Change Updates Mode +The script sends messages in the following format: + +``` +đ **13-01-25 new muted tests in `main` for [team-name](https://github.com/orgs/ydb-platform/teams/team-name)** #teamname + + đ¯ `Issue Title` [#12345](https://github.com/...) + đ¯ `Another Issue Title` [#12346](https://github.com/...) + +đ **[Total muted tests: 150](https://datalens.yandex/4un3zdm0zcnyr?owner_team=team-name) đ´+5 muted /đĸ-2 unmuted** + +fyi: @user1 @user2 +``` + +**Statistics explanation:** +- `đ **[Total muted tests: N](dashboard_url) đ´+M muted /đĸ-K unmuted**` - Total muted tests with today's changes +- `đ **[Total muted tests: N](dashboard_url) đ´+M muted**` - Total muted tests with today's additions only +- `đ **[Total muted tests: N](dashboard_url) đĸ-K unmuted**` - Total muted tests with today's unmutes only +- `đ **[Total muted tests: N](dashboard_url)**` - Total muted tests (no changes today) + +### Periodic Trend Updates Mode +The script sends trend reports in the following format: + +``` +đ **Week Over Week changes for team [team-name](https://github.com/orgs/ydb-platform/teams/team-name)** #teamname + +đ **[Total muted tests: 150](https://datalens.yandex/4un3zdm0zcnyr?owner_team=team-name) (đ´+10 vs 7 days ago)** + +fyi: @user1 @user2 + +Chart shows muted tests trend over the last 30 days. +``` + +**Period statistics explanation:** +- `(đ´+N vs X days ago)` - Increase compared to previous period +- `(đĸ-N vs X days ago)` - Decrease compared to previous period +- `(0 vs X days ago)` - No change compared to previous period + +## Team Channels Configuration + +The `--team-channels` parameter expects a JSON configuration: + +```json +{ + "default_channel": "default-channel-name", + "teams": { + "team-name": { + "responsible": ["@user1", "@user2"], + "channel": "specific-channel-name" + }, + "another-team": { + "responsible": "@single-user", + "channel": "another-channel" + } + }, + "channels": { + "default-channel-name": "123456789/1", + "specific-channel-name": "987654321/2", + "another-channel": "555666777" + } +} +``` + +## Environment Variables + +- `TELEGRAM_BOT_TOKEN` - Telegram bot token +- `TEAM_CHANNELS` - Team channels configuration JSON +- `YDB_ENDPOINT` - YDB database endpoint +- `YDB_DATABASE` - YDB database path +- `YDB_SERVICE_ACCOUNT_KEY_FILE_CREDENTIALS` - Path to YDB credentials file + +## Team Blacklist + +The script supports a blacklist for periodic updates (weekly/monthly). Teams in the blacklist will be skipped during periodic trend updates but will still receive immediate mute-change notifications. + +To add teams to the blacklist, edit the `PERIOD_UPDATE_BLACKLIST` constant in the script: + +```python +# Teams blacklisted from weekly/monthly updates +PERIOD_UPDATE_BLACKLIST = { + 'storage', # Example: storage team + 'team-name' # Add more teams as needed +} +``` + +**Note:** The blacklist only affects periodic updates (`--period-update` mode), not immediate mute-change notifications (`--on-mute-change-update` mode). + +## Dependencies + +- `ydb` - YDB Python client library +- `requests` - HTTP library for Telegram API +- `telegram` - Telegram bot functionality (from send_telegram_message.py) +- `matplotlib` - For creating trend plots (optional, only if --include-plots is used) + +## Examples with Plots + +### On-Mute-Change Updates with Plots +```bash +# Send messages with trend plots +python .github/scripts/telegram/parse_and_send_team_issues.py \ + --on-mute-change-update \ + --file "formatted_results.txt" \ + --team-channels '{"teams": {"team-name": {"responsible": ["@user1"], "channel": "channel-name"}}, "channels": {"channel-name": "123456789/1"}}' \ + --bot-token "YOUR_BOT_TOKEN" \ + --include-plots \ + --use-yesterday + +# Send messages with debug plots saved to custom directory +python .github/scripts/telegram/parse_and_send_team_issues.py \ + --on-mute-change-update \ + --file "formatted_results.txt" \ + --team-channels '{"teams": {"team-name": {"responsible": ["@user1"], "channel": "channel-name"}}, "channels": {"channel-name": "123456789/1"}}' \ + --bot-token "YOUR_BOT_TOKEN" \ + --include-plots \ + --debug-plots-dir "/path/to/debug/plots" +``` + +### Periodic Trend Updates (Always Include Plots) +```bash +# Weekly trend updates with debug plots +python .github/scripts/telegram/parse_and_send_team_issues.py \ + --period-update week \ + --team-channels '{"default_channel": "main_channel", "teams": {"team-name": {"responsible": ["@user1"], "channel": "main_channel"}}, "channels": {"main_channel": "123456789/1"}}' \ + --bot-token "YOUR_BOT_TOKEN" \ + --ydb-endpoint "grpcs://ydb.serverless.yandexcloud.net:2135" \ + --ydb-database "/ru-central1/b1g8ejbrie0sfh5k0j2j/etn8l4e3hbti8k4n5g2e" \ + --debug-plots-dir "/path/to/debug/plots" + +# Monthly trend updates (dry run) +python .github/scripts/telegram/parse_and_send_team_issues.py \ + --period-update month \ + --team-channels '{"default_channel": "main_channel", "teams": {"team-name": {"responsible": ["@user1"], "channel": "main_channel"}}, "channels": {"main_channel": "123456789/1"}}' \ + --bot-token "YOUR_BOT_TOKEN" \ + --ydb-endpoint "grpcs://ydb.serverless.yandexcloud.net:2135" \ + --ydb-database "/ru-central1/b1g8ejbrie0sfh5k0j2j/etn8l4e3hbti8k4n5g2e" \ + --dry-run +``` diff --git a/.github/scripts/telegram/README_team_issues.md b/.github/scripts/telegram/README_team_issues.md index 363d59e9819..20fc0d5e743 100644 --- a/.github/scripts/telegram/README_team_issues.md +++ b/.github/scripts/telegram/README_team_issues.md @@ -25,13 +25,14 @@ python .github/scripts/telegram/parse_and_send_team_issues.py \ ## Message Format ``` -đ **07-09-24 new muted tests for [team-name](https://github.com/orgs/ydb-platform/teams/team-name)** #team-name +đ **07-09-24 new muted tests in `main` for [team-name](https://github.com/orgs/ydb-platform/teams/team-name)** #teamname -fyi: @responsible1 @responsible2 + đ¯ `Issue Title` [#12345](https://github.com/...) + đ¯ `Another Issue Title` [#12346](https://github.com/...) - - đ¯ [Issue URL](Issue URL) - `Issue Title` - - đ¯ [Issue URL](Issue URL) - `Issue Title` +đ **[Total muted tests: 150](https://datalens.yandex/4un3zdm0zcnyr?owner_team=team-name) đ´+5 muted /đĸ-2 unmuted** +fyi: @responsible1 @responsible2 ``` ## Team Channels Configuration diff --git a/.github/scripts/telegram/README_telegram.md b/.github/scripts/telegram/README_telegram.md index 89b5bbdd554..4512d514188 100644 --- a/.github/scripts/telegram/README_telegram.md +++ b/.github/scripts/telegram/README_telegram.md @@ -16,6 +16,13 @@ python .github/scripts/telegram/send_telegram_message.py \ --message "Hello, World!" \ --bot-token "YOUR_BOT_TOKEN" \ --chat-id "1234567890" + +# Send photo with caption +python .github/scripts/telegram/send_telegram_message.py \ + --message "Check out this image!" \ + --photo "path/to/image.jpg" \ + --bot-token "YOUR_BOT_TOKEN" \ + --chat-id "1234567890" ``` ## Parameters @@ -23,6 +30,7 @@ python .github/scripts/telegram/send_telegram_message.py \ - `--bot-token` - Telegram bot token (required) - `--chat-id` - Chat/channel ID (required) - `--message` - Message text or path to file to send (automatically detected) +- `--photo` - Path to photo file to send (optional) - `--parse-mode` - Parse mode (Markdown, HTML, None) - default: Markdown - `--delay` - Delay between messages in seconds - default: 1 - `--max-length` - Maximum message length - default: 4000 @@ -35,6 +43,7 @@ python .github/scripts/telegram/send_telegram_message.py \ - â
Automatic splitting of long messages into chunks - â
Markdown formatting support +- â
Photo sending with caption support - â
Delay between messages to avoid API limits - â
Message thread support - â
Disable link previews diff --git a/.github/scripts/telegram/parse_and_send_team_issues.py b/.github/scripts/telegram/parse_and_send_team_issues.py index 0a3e68ff5e0..04a3047d9ee 100644 --- a/.github/scripts/telegram/parse_and_send_team_issues.py +++ b/.github/scripts/telegram/parse_and_send_team_issues.py @@ -10,10 +10,420 @@ import requests import argparse import re import json -from datetime import datetime +import configparser +import tempfile +import base64 +import shutil +from datetime import datetime, timedelta from pathlib import Path from send_telegram_message import send_telegram_message +try: + import ydb + YDB_AVAILABLE = True +except ImportError: + YDB_AVAILABLE = False + print("â ī¸ YDB client not available. Install with: pip install ydb") + +try: + import matplotlib.pyplot as plt + import matplotlib.dates as mdates + import numpy as np + MATPLOTLIB_AVAILABLE = True +except ImportError: + MATPLOTLIB_AVAILABLE = False + print("â ī¸ Matplotlib not available. Install with: pip install matplotlib") + +# Configuration constants +MUTE_UPDATE_SHOW_DIFF = False # Set to True to show +/- statistics in mute update messages + +# Teams blacklisted from weekly/monthly updates +PERIOD_UPDATE_BLACKLIST = { + 'storage' # Add team names that should not receive periodic updates +} + +# URL constants (only for duplicated URLs) +GITHUB_ORG_TEAMS_URL = "https://github.com/orgs/ydb-platform/teams" +DATALENS_DASHBOARD_URL = "https://datalens.yandex/4un3zdm0zcnyr?owner_team={team_name}" + + +def _setup_ydb_connection(database_endpoint=None, database_path=None, credentials_path=None): + """ + Set up YDB connection parameters and credentials. + + Args: + database_endpoint (str): YDB database endpoint + database_path (str): YDB database path + credentials_path (str): Path to service account credentials JSON file + + Returns: + tuple: (database_endpoint, database_path) or (None, None) if error + """ + if not YDB_AVAILABLE: + print("â YDB client not available") + return None, None + + try: + # Set up credentials if provided + if credentials_path and os.path.exists(credentials_path): + os.environ["YDB_SERVICE_ACCOUNT_KEY_FILE_CREDENTIALS"] = credentials_path + elif "CI_YDB_SERVICE_ACCOUNT_KEY_FILE_CREDENTIALS" in os.environ: + os.environ["YDB_SERVICE_ACCOUNT_KEY_FILE_CREDENTIALS"] = os.environ["CI_YDB_SERVICE_ACCOUNT_KEY_FILE_CREDENTIALS"] + + # Try to load from config file if not provided + if not database_endpoint or not database_path: + try: + config = configparser.ConfigParser() + config_file_path = os.path.join(os.path.dirname(__file__), "../../config/ydb_qa_db.ini") + if os.path.exists(config_file_path): + config.read(config_file_path) + if not database_endpoint and "QA_DB" in config and "DATABASE_ENDPOINT" in config["QA_DB"]: + database_endpoint = config["QA_DB"]["DATABASE_ENDPOINT"] + if not database_path and "QA_DB" in config and "DATABASE_PATH" in config["QA_DB"]: + database_path = config["QA_DB"]["DATABASE_PATH"] + except Exception as e: + print(f"â ī¸ Could not load config file: {e}") + + # Use default values if still not provided + if not database_endpoint: + database_endpoint = os.getenv('YDB_ENDPOINT', 'grpcs://ydb.serverless.yandexcloud.net:2135') + if not database_path: + database_path = os.getenv('YDB_DATABASE', '/ru-central1/b1g8ejbrie0sfh5k0j2j/etn8l4e3hbti8k4n5g2e') + + return database_endpoint, database_path + + except Exception as e: + print(f"â Error setting up YDB connection: {e}") + return None, None + + +def _execute_ydb_query(database_endpoint, database_path, query, description): + """ + Execute a YDB query and return results. + + Args: + database_endpoint (str): YDB database endpoint + database_path (str): YDB database path + query (str): SQL query to execute + description (str): Description for logging + + Returns: + list: Query results or None if error + """ + try: + print(f"đ {description}") + + with ydb.Driver( + endpoint=database_endpoint, + database=database_path, + credentials=ydb.credentials_from_env_variables(), + ) as driver: + driver.wait(timeout=10, fail_fast=True) + print("â
Successfully connected to YDB") + + table_client = ydb.TableClient(driver, ydb.TableClientSettings()) + query_obj = ydb.ScanQuery(query, {}) + it = table_client.scan_query(query_obj) + results = [] + for result in it: + results.extend(result.result_set.rows) + + print(f"đ Query returned {len(results)} rows") + return results + + except Exception as e: + print(f"â Error executing YDB query: {e}") + print(f"â Error type: {type(e).__name__}") + import traceback + print(f"â Traceback: {traceback.format_exc()}") + return None + + +def get_all_team_data(database_endpoint=None, database_path=None, credentials_path=None, use_yesterday=False): + """ + Get all team data (stats + trends) from YDB in one optimized query. + + Args: + database_endpoint (str): YDB database endpoint + database_path (str): YDB database path + credentials_path (str): Path to service account credentials JSON file + use_yesterday (bool): If True, use yesterday's data for development convenience + + Returns: + dict: Dictionary with team names as keys and their data, or None if error + """ + # Set up connection + print(f"đ Setting up YDB connection:") + print(f" database_endpoint: {database_endpoint}") + print(f" database_path: {database_path}") + print(f" credentials_path: {credentials_path}") + + endpoint, path = _setup_ydb_connection(database_endpoint, database_path, credentials_path) + if not endpoint or not path: + print("â Failed to set up YDB connection") + return None + + print(f"â
YDB connection configured: {endpoint} / {path}") + + # Calculate target date + if use_yesterday: + target_date = datetime.now() - timedelta(days=1) + else: + target_date = datetime.now() + + yesterday_date = target_date - timedelta(days=1) + start_date = target_date - timedelta(days=30) + + print(f"đ Date calculation:") + print(f" Current datetime.now(): {datetime.now()}") + print(f" use_yesterday: {use_yesterday}") + print(f" target_date: {target_date}") + print(f" start_date: {start_date}") + print(f" yesterday_date: {yesterday_date}") + + # Single optimized query for all data + all_data_query = f""" + SELECT + owner, + date_window, + COUNT(*) as daily_count, + SUM(CASE WHEN mute_state_change_date = Date('{target_date.strftime('%Y-%m-%d')}') THEN 1 ELSE 0 END) as today_count + FROM `test_results/analytics/test_muted_monitor_mart` + WHERE date_window >= Date('{start_date.strftime('%Y-%m-%d')}') + AND date_window <= Date('{target_date.strftime('%Y-%m-%d')}') + AND is_muted = 1 + AND branch = 'main' + AND build_type = 'relwithdebinfo' + GROUP BY owner, date_window + ORDER BY owner, date_window + """ + + # Execute query + print(f"đ Query details:") + print(f" Start date: {start_date.strftime('%Y-%m-%d')}") + print(f" Target date: {target_date.strftime('%Y-%m-%d')}") + print(f" Endpoint: {endpoint}") + print(f" Path: {path}") + + results = _execute_ydb_query(endpoint, path, all_data_query, f"Getting all team data from {start_date.strftime('%Y-%m-%d')} to {target_date.strftime('%Y-%m-%d')}") + if results is None: + return None + + # Process results + team_data = {} + base_date = datetime(1970, 1, 1) + + for row in results: + owner = row.owner + if not owner: + continue + + # Handle both "TEAM:@ydb-platform/teamname" and "Unknown" formats + if owner.startswith('TEAM:@ydb-platform/'): + team_name = owner.split('/')[-1] + elif owner == 'Unknown': + team_name = 'Unknown' + else: + # Skip other formats + continue + + if team_name not in team_data: + team_data[team_name] = { + 'stats': {'total': 0, 'today': 0, 'minus_today': 0}, + 'trend': {} + } + + # Convert days since epoch to date + date_obj = base_date + timedelta(days=row.date_window) + date_str = date_obj.strftime('%Y-%m-%d') + + # Add to trend data + team_data[team_name]['trend'][date_str] = row.daily_count + + # Update stats for target date + if date_str == target_date.strftime('%Y-%m-%d'): + team_data[team_name]['stats']['total'] = row.daily_count + team_data[team_name]['stats']['today'] = row.today_count + + # Calculate "minus today" for each team and fix total if needed + for team_name, data in team_data.items(): + trend = data['trend'] + yesterday_str = yesterday_date.strftime('%Y-%m-%d') + + # If total is 0 but we have trend data, use the latest available value + if data['stats']['total'] == 0 and trend: + latest_date = max(trend.keys()) + data['stats']['total'] = trend[latest_date] + print(f"đ Fixed total for team {team_name}: using {latest_date} value {trend[latest_date]}") + + if yesterday_str in trend: + yesterday_total = trend[yesterday_str] + today_total = data['stats']['total'] + today_new = data['stats']['today'] + + # minus_today = yesterday_total - (today_total - today_new) + data['stats']['minus_today'] = max(0, yesterday_total - (today_total - today_new)) + + print(f"đ Processed data for {len(team_data)} teams") + return team_data + + +def get_muted_tests_stats(database_endpoint=None, database_path=None, credentials_path=None, use_yesterday=False): + """ + Get statistics about muted tests from YDB by team. + + Args: + database_endpoint (str): YDB database endpoint + database_path (str): YDB database path + credentials_path (str): Path to service account credentials JSON file + use_yesterday (bool): If True, use yesterday's data for development convenience + + Returns: + dict: Dictionary with team names as keys and {'total': count, 'today': count} as values, or None if error + """ + + # Use the optimized function to get all data + all_data = get_all_team_data(database_endpoint, database_path, credentials_path, use_yesterday) + if all_data is None: + return None + + # Extract just the stats part + team_stats = {} + for team_name, data in all_data.items(): + team_stats[team_name] = data['stats'] + + print(f"đ Found statistics for {len(team_stats)} teams") + return team_stats + + +def get_monthly_trend_data(database_endpoint=None, database_path=None, credentials_path=None, team_name=None, use_yesterday=False): + """ + Get monthly trend data for a specific team. + + Args: + database_endpoint (str): YDB database endpoint + database_path (str): YDB database path + credentials_path (str): Path to service account credentials JSON file + team_name (str): Team name to get data for + use_yesterday (bool): If True, use yesterday as end date for development convenience + + Returns: + dict: Dictionary with dates as keys and counts as values, or None if error + """ + # Use the optimized function to get all data + all_data = get_all_team_data(database_endpoint, database_path, credentials_path, use_yesterday) + if all_data is None: + return None + + # Extract trend data for the specific team + if team_name in all_data: + trend_data = all_data[team_name]['trend'] + print(f"đ Found trend data for {len(trend_data)} days for team '{team_name}'") + return trend_data + else: + print(f"â ī¸ No data found for team '{team_name}'") + return None + + +def create_trend_plot(team_name, trend_data, debug_dir=None): + """ + Create a trend plot for muted tests. + + Args: + team_name (str): Team name + trend_data (dict): Dictionary with dates as keys and counts as values + debug_dir (str): Directory to save debug plot files (if None, debug mode is disabled) + + Returns: + str: Base64 encoded image data, or None if error + """ + if not MATPLOTLIB_AVAILABLE: + print("â Matplotlib not available for plotting") + return None + + if not trend_data: + print("â ī¸ No trend data available for plotting") + return None + + try: + # Prepare data + dates = [] + counts = [] + for date_str in sorted(trend_data.keys()): + dates.append(datetime.strptime(date_str, '%Y-%m-%d')) + counts.append(trend_data[date_str]) + + # Create plot + plt.figure(figsize=(10, 6)) + plt.plot(dates, counts, marker='o', linewidth=2, markersize=4) + plt.title(f'Muted Tests Trend - {team_name}', fontsize=14, fontweight='bold') + plt.xlabel('Date', fontsize=12) + plt.ylabel('Number of Muted Tests', fontsize=12) + plt.grid(True, alpha=0.3) + plt.ylim(bottom=0) # Start y-axis from 0 + + # Format x-axis + plt.gca().xaxis.set_major_formatter(mdates.DateFormatter('%m-%d')) + plt.gca().xaxis.set_major_locator(mdates.DayLocator(interval=3)) + plt.xticks(rotation=45) + + # Add trend line + if len(dates) > 1: + x_numeric = np.arange(len(dates)) + z = np.polyfit(x_numeric, counts, 1) + p = np.poly1d(z) + plt.plot(dates, p(x_numeric), "r--", alpha=0.8, linewidth=2, label='Trend') + plt.legend() + + # Add annotation for the last point + if dates and counts: + last_date = dates[-1] + last_count = counts[-1] + plt.annotate(f'{last_count}', + xy=(last_date, last_count), + xytext=(10, 10), + textcoords='offset points', + bbox=dict(boxstyle='round,pad=0.3', facecolor='yellow', alpha=0.7), + arrowprops=dict(arrowstyle='->', connectionstyle='arc3,rad=0'), + fontsize=10, fontweight='bold') + + plt.tight_layout() + + # Save to temporary file + with tempfile.NamedTemporaryFile(suffix='.png', delete=False) as tmp_file: + plt.savefig(tmp_file.name, dpi=150, bbox_inches='tight') + tmp_path = tmp_file.name + + print(f"đ Created trend plot for {team_name}: {tmp_path}") + print(f"đ File size: {os.path.getsize(tmp_path)} bytes") + + # Save to debug directory if requested + if debug_dir: + os.makedirs(debug_dir, exist_ok=True) + debug_path = os.path.join(debug_dir, f"trend_{team_name}_{datetime.now().strftime('%Y%m%d_%H%M%S')}.png") + + # Copy to debug directory + shutil.copy2(tmp_path, debug_path) + print(f"đ Debug plot saved to: {debug_path}") + + # Read and encode as base64 + with open(tmp_path, 'rb') as f: + image_data = f.read() + + print(f"đ Base64 data length: {len(base64.b64encode(image_data).decode('utf-8'))} characters") + + # Clean up temporary file (but keep debug file) + os.unlink(tmp_path) + plt.close() + + # Encode as base64 + base64_data = base64.b64encode(image_data).decode('utf-8') + return base64_data + + except Exception as e: + print(f"â Error creating trend plot: {e}") + return None + def parse_team_issues(content): """ @@ -65,7 +475,7 @@ def parse_team_issues(content): def escape_markdown(text): """ - Escape special Markdown characters for Telegram, but preserve link structure. + Escape special MarkdownV2 characters for Telegram, but preserve bold formatting and link structure. Args: text (str): Text to escape @@ -73,14 +483,71 @@ def escape_markdown(text): Returns: str: Escaped text """ - special_chars = ['*', '~', '>', '#', '=', '|', '{', '}', '!'] + # For MarkdownV2, we need to escape these characters: _ * [ ] ( ) ~ ` > # + - = | { } . ! + # But we want to preserve * for bold formatting, [ ] ( ) for links, and content inside backticks + + # First, protect inline code by temporarily replacing them + code_pattern = r'`([^`]+)`' + code_blocks = [] + + def replace_code(match): + code_blocks.append(match.group(1)) + return f"CODEPLACEHOLDER{len(code_blocks)-1}N" + + # Replace all inline code with placeholders + text = re.sub(code_pattern, replace_code, text) + + # Then protect links by temporarily replacing them + link_pattern = r'\[([^\]]+)\]\(([^)]+)\)' + links = [] + + def replace_link(match): + links.append((match.group(1), match.group(2))) + return f"LINKPLACEHOLDER{len(links)-1}N" + + # Replace all links with placeholders + text = re.sub(link_pattern, replace_link, text) + + # Also protect standalone URLs (http/https) by temporarily replacing them + url_pattern = r'https?://[^\s\)]+' + urls = [] + + def replace_url(match): + urls.append(match.group(0)) + return f"URLPLACEHOLDER{len(urls)-1}N" + + # Replace all standalone URLs with placeholders + text = re.sub(url_pattern, replace_url, text) + + # Escape special characters for MarkdownV2 (single backslash) + # Characters that need escaping: _ * [ ] ( ) ~ ` > # + - = | { } . ! + special_chars = ['_', '~', '`', '>', '#', '+', '-', '=', '|', '{', '}', '.', '!', '(', ')'] for char in special_chars: text = text.replace(char, f'\\{char}') + + # Restore standalone URLs (they should NOT be escaped) + for i, url in enumerate(urls): + text = text.replace(f"URLPLACEHOLDER{i}N", url) + + # Restore links, escaping special characters in link text but NOT in URLs + for i, (link_text, link_url) in enumerate(links): + # Escape special characters in link text only + escaped_text = link_text + for char in special_chars: + escaped_text = escaped_text.replace(char, f'\\{char}') + + # URLs should NOT be escaped (Telegram handles them correctly) + text = text.replace(f"LINKPLACEHOLDER{i}N", f"[{escaped_text}]({link_url})") + + # Restore inline code (content should NOT be escaped) + for i, code_content in enumerate(code_blocks): + text = text.replace(f"CODEPLACEHOLDER{i}N", f"`{code_content}`") + return text -def format_team_message(team_name, issues, team_responsible=None): +def format_team_message(team_name, issues, team_responsible=None, muted_stats=None, show_diff=False): """ Format message for a specific team. @@ -88,6 +555,8 @@ def format_team_message(team_name, issues, team_responsible=None): team_name (str): Team name issues (list): List of issues for the team team_responsible (dict): Dictionary mapping team names to responsible usernames + muted_stats (dict): Dictionary with team names as keys and {'total': count, 'today': count} as values + show_diff (bool): Whether to show +/- statistics Returns: str: Formatted message @@ -100,9 +569,48 @@ def format_team_message(team_name, issues, team_responsible=None): # Start with title and team tag (replace - with _ in tag) team_tag = team_name.replace('-', '') - message = f"đ **{current_date} new muted tests for [{team_name}](https://github.com/orgs/ydb-platform/teams/{team_name})** #{team_tag}\n\n" + # Create team URL and escape it + team_url = f"{GITHUB_ORG_TEAMS_URL}/{team_name}" + message = f"đ *{current_date} new muted tests in `main` for [{team_name}]({team_url})* #{team_tag}\n\n" + + for issue in issues: + # Extract issue number from URL for compact display + issue_number = issue['url'].split('/')[-1] if '/' in issue['url'] else issue['url'] + + # Remove "in main" from title if present and "Mute " prefix + title = issue['title'] + if title.endswith(' in main'): + title = title[:-8] # Remove " in main" + if title.startswith('Mute '): + title = title[5:] # Remove "Mute " prefix + + # Wrap title in backticks (will be escaped later with the whole message) + message += f" đ¯ `{title}` [#{issue_number}]({issue['url']})\n" - # Add responsible users on new line with "fyi:" prefix + # Add muted tests statistics for this specific team if available (moved to end) + if muted_stats and team_name in muted_stats: + team_stats = muted_stats[team_name] + total = team_stats['total'] + today = team_stats['today'] + minus_today = team_stats.get('minus_today', 0) + + # Create dashboard URL for the team + dashboard_url = DATALENS_DASHBOARD_URL.format(team_name=team_name) + + # Format statistics with color coding and emojis + if show_diff: + if today > 0 and minus_today > 0: + message += f"\nđ *[Total muted tests: {total}]({dashboard_url}) đ´+{today} muted /đĸ-{minus_today} unmuted*" + elif today > 0: + message += f"\nđ *[Total muted tests: {total}]({dashboard_url}) đ´+{today} muted*" + elif minus_today > 0: + message += f"\nđ *[Total muted tests: {total}]({dashboard_url}) đĸ-{minus_today} unmuted*" + else: + message += f"\nđ *[Total muted tests: {total}]({dashboard_url})*" + else: + message += f"\nđ *[Total muted tests: {total}]({dashboard_url})*" + + # Add responsible users on new line with "fyi:" prefix (moved after statistics) if team_responsible and team_name in team_responsible: responsible = team_responsible[team_name] # Handle both single responsible and list of responsibles @@ -110,12 +618,7 @@ def format_team_message(team_name, issues, team_responsible=None): responsible_str = " ".join(f"@{r.replace('_', '\\_')}" if not r.startswith('@') else r.replace('_', '\\_') for r in responsible) else: responsible_str = f"@{responsible.replace('_', '\\_')}" if not responsible.startswith('@') else responsible.replace('_', '\\_') - message += f"fyi: {responsible_str}\n\n" - - for issue in issues: - # Escape the title for Markdown and wrap in backticks - escaped_title = escape_markdown(issue['title']) - message += f" - đ¯ [{issue['url']}]({issue['url']}) - `{escaped_title}`\n" + message += f"\n\nfyi: {responsible_str}" # Add empty line at the end for better readability message += "\n" @@ -205,7 +708,7 @@ def get_team_config(team_name, team_channels): return None, None, None -def send_team_messages(teams, bot_token, delay=2, max_retries=5, retry_delay=10, team_channels=None, dry_run=False): +def send_team_messages(teams, bot_token, delay=2, max_retries=5, retry_delay=10, team_channels=None, dry_run=False, muted_stats=None, include_plots=False, ydb_config=None, debug_plots_dir=None, all_team_data=None, show_diff=False): """ Send separate messages for each team. @@ -217,6 +720,12 @@ def send_team_messages(teams, bot_token, delay=2, max_retries=5, retry_delay=10, retry_delay (int): Delay between retry attempts in seconds team_channels (dict): Dictionary mapping team names to their specific channel configs dry_run (bool): If True, only print messages without sending to Telegram + muted_stats (dict): Dictionary with team names as keys and {'total': count, 'today': count} as values + include_plots (bool): If True, include trend plots in messages + ydb_config (dict): YDB configuration for trend data + debug_plots_dir (str): Directory to save debug plot files (if None, debug mode is disabled) + all_team_data (dict): Pre-fetched team data to avoid repeated queries + show_diff (bool): Whether to show +/- statistics in messages """ total_teams = len(teams) @@ -241,11 +750,20 @@ def send_team_messages(teams, bot_token, delay=2, max_retries=5, retry_delay=10, continue # Format message - message = format_team_message(team_name, issues, team_responsible) + message = format_team_message(team_name, issues, team_responsible, muted_stats, show_diff) if not message.strip(): continue + # Escape the entire message for MarkdownV2 + message = escape_markdown(message) + + # Print final message before sending + print(f"đ Final message for {team_name}:") + print("-" * 80) + print(message) + print("-" * 80) + if dry_run: print(f"\n--- Team: {team_name} ---") print(f"đ¨ Channel: {team_chat_id}" + (f" (thread {team_thread_id})" if team_thread_id else "")) @@ -254,11 +772,86 @@ def send_team_messages(teams, bot_token, delay=2, max_retries=5, retry_delay=10, else: print(f"đ¨ Sending message for team: {team_name} ({len(issues)} issues)") - if send_telegram_message(bot_token, team_chat_id, message, "Markdown", team_thread_id, True, max_retries, retry_delay): - sent_count += 1 - print(f"â
Message sent for team: {team_name}") + # Get trend plot if requested + plot_data = None + print(f"đ Plot settings: include_plots={include_plots}, ydb_config={ydb_config is not None}, MATPLOTLIB_AVAILABLE={MATPLOTLIB_AVAILABLE}") + + if include_plots and MATPLOTLIB_AVAILABLE: + # Use pre-fetched data if available, otherwise fetch on demand + if all_team_data and team_name in all_team_data: + trend_data = all_team_data[team_name]['trend'] + print(f"đ Using cached trend data for {team_name}: {len(trend_data)} days") + elif ydb_config: + print(f"đ Getting trend data for team: {team_name}") + trend_data = get_monthly_trend_data( + database_endpoint=ydb_config.get('endpoint'), + database_path=ydb_config.get('path'), + credentials_path=ydb_config.get('credentials'), + team_name=team_name, + use_yesterday=ydb_config.get('use_yesterday', False) + ) + else: + trend_data = None + + if trend_data: + plot_data = create_trend_plot(team_name, trend_data, debug_plots_dir) + if plot_data: + print(f"đ Created trend plot for team: {team_name} (data length: {len(plot_data)})") + else: + print(f"â ī¸ Could not create trend plot for team: {team_name}") + else: + print(f"â ī¸ No trend data available for team: {team_name}") + elif include_plots and not MATPLOTLIB_AVAILABLE: + print(f"â ī¸ Matplotlib not available, skipping plot for team: {team_name}") + elif include_plots and not ydb_config: + print(f"â ī¸ YDB config not available, skipping plot for team: {team_name}") + else: + print(f"âšī¸ Plots disabled for team: {team_name}") + + # Send message with or without plot + if plot_data: + # Save plot to temporary file and send as photo + with tempfile.NamedTemporaryFile(suffix='.png', delete=False) as tmp_file: + tmp_file.write(base64.b64decode(plot_data)) + tmp_path = tmp_file.name + + print(f"đ Saved plot to temporary file: {tmp_path}") + print(f"đ File size: {os.path.getsize(tmp_path)} bytes") + + # Also save to debug directory for final file if requested + if debug_plots_dir: + os.makedirs(debug_plots_dir, exist_ok=True) + final_debug_path = os.path.join(debug_plots_dir, f"final_{team_name}_{datetime.now().strftime('%Y%m%d_%H%M%S')}.png") + shutil.copy2(tmp_path, final_debug_path) + print(f"đ Final debug plot saved to: {final_debug_path}") + + try: + print(f"đ¤ Sending message with plot for team: {team_name}") + print(f"đ¤ Chat ID: {team_chat_id}, Thread ID: {team_thread_id}") + print(f"đ¤ Photo path: {tmp_path}") + print(f"đ¤ Message length: {len(message)} characters") + + if send_telegram_message(bot_token, team_chat_id, message, "MarkdownV2", team_thread_id, True, max_retries, retry_delay, photo_path=tmp_path): + sent_count += 1 + print(f"â
Message with plot sent for team: {team_name}") + else: + print(f"â Failed to send message with plot for team: {team_name} after {max_retries} retries") + finally: + # Clean up temporary file + if os.path.exists(tmp_path): + os.unlink(tmp_path) + print(f"đī¸ Cleaned up temporary file: {tmp_path}") else: - print(f"â Failed to send message for team: {team_name} after {max_retries} retries") + # Send regular message + print(f"đ¤ Sending regular message for team: {team_name}") + print(f"đ¤ Chat ID: {team_chat_id}, Thread ID: {team_thread_id}") + print(f"đ¤ Message length: {len(message)} characters") + + if send_telegram_message(bot_token, team_chat_id, message, "MarkdownV2", team_thread_id, True, max_retries, retry_delay): + sent_count += 1 + print(f"â
Message sent for team: {team_name}") + else: + print(f"â Failed to send message for team: {team_name} after {max_retries} retries") # Add delay between messages if sent_count < total_teams: @@ -372,11 +965,248 @@ def load_team_channels(team_channels_json): return None +def send_period_updates(period, bot_token, team_channels, ydb_config, delay=2, max_retries=5, retry_delay=10, dry_run=False, debug_plots_dir=None): + """ + Send periodic trend updates for all teams. + + Args: + period (str): Period type ('week' or 'month') + bot_token (str): Telegram bot token + team_channels (dict): Dictionary mapping team names to their channel configurations + ydb_config (dict): YDB configuration + delay (int): Delay between messages in seconds + max_retries (int): Maximum number of retry attempts + retry_delay (int): Delay between retry attempts in seconds + dry_run (bool): If True, only print messages without sending to Telegram + debug_plots_dir (str): Directory to save debug plot files + + Returns: + bool: True if successful, False otherwise + """ + print(f"đ Starting {period}ly trend updates...") + + # Get all team data for trends + all_team_data = get_all_team_data( + database_endpoint=ydb_config.get('endpoint'), + database_path=ydb_config.get('path'), + credentials_path=ydb_config.get('credentials'), + use_yesterday=ydb_config.get('use_yesterday', False) + ) + + if not all_team_data: + print("â Could not fetch team data for trend updates") + return False + + # Get teams from data (all teams that have data) + teams_from_data = list(all_team_data.keys()) + if not teams_from_data: + print("â No teams found in data") + return False + + # Filter out blacklisted teams + teams_to_process = [team for team in teams_from_data if team not in PERIOD_UPDATE_BLACKLIST] + blacklisted_count = len(teams_from_data) - len(teams_to_process) + + if blacklisted_count > 0: + print(f"âī¸ Skipping {blacklisted_count} blacklisted teams: {', '.join(team for team in teams_from_data if team in PERIOD_UPDATE_BLACKLIST)}") + + print(f"đ¤ Sending {period}ly updates for {len(teams_to_process)} teams from data...") + + success_count = 0 + total_teams = len(teams_to_process) + + for team_name in teams_to_process: + + # Get team channel configuration + team_responsible, team_chat_id, team_thread_id = get_team_config(team_name, team_channels) + + # If team not found in config, use default channel + if not team_chat_id and team_channels: + default_channel_name = team_channels.get('default_channel') + if default_channel_name and 'channels' in team_channels: + if default_channel_name in team_channels['channels']: + team_chat_id, team_thread_id = parse_chat_and_thread_id(team_channels['channels'][default_channel_name]) + print(f"đ¨ Using default channel '{default_channel_name}' for team {team_name}: {team_chat_id}") + + # Determine channel name for logging + if team_channels and 'teams' in team_channels and team_name in team_channels['teams']: + team_config = team_channels['teams'][team_name] + team_channel_name = team_config.get('channel', team_channels.get('default_channel', 'default')) + else: + team_channel_name = team_channels.get('default_channel', 'default') if team_channels else 'default' + + if not team_chat_id: + print(f"â ī¸ No channel configuration for team: {team_name} (skipping)") + continue + + # Create trend message + period_title = "Weekly Muted Tests Report" if period == "week" else "Monthly Muted Tests Report" + team_tag = team_name.replace('-', '') + team_url = f"{GITHUB_ORG_TEAMS_URL}/{team_name}" + message = f"đ *{period_title}* for team [{team_name}]({team_url}) #{team_tag}\n\n" + + # Add trend statistics if available + if team_name in all_team_data: + team_stats = all_team_data[team_name]['stats'] + trend_data = all_team_data[team_name]['trend'] + total = team_stats['total'] + dashboard_url = DATALENS_DASHBOARD_URL.format(team_name=team_name) + + # Calculate period change + period_days = 7 if period == "week" else 30 + # Use the same date logic as in get_all_team_data + current_date = datetime.now() - timedelta(days=1) if ydb_config.get('use_yesterday', False) else datetime.now() + previous_date = current_date - timedelta(days=period_days) + + current_date_str = current_date.strftime('%Y-%m-%d') + previous_date_str = previous_date.strftime('%Y-%m-%d') + + current_count = trend_data.get(current_date_str, 0) + previous_count = trend_data.get(previous_date_str, 0) + + # Check if we have data for both dates to calculate meaningful change + has_current_data = current_date_str in trend_data + has_previous_data = previous_date_str in trend_data + + # Debug information + print(f"đ Debug for team {team_name}:") + print(f" Current date: {current_date_str}, count: {current_count}, has_data: {has_current_data}") + print(f" Previous date: {previous_date_str}, count: {previous_count}, has_data: {has_previous_data}") + print(f" Available dates in trend_data: {sorted(trend_data.keys())[-5:]}") # Last 5 dates + + message += f"đ *[Total muted tests: {total}]({dashboard_url})*\n\n" + + # Add change information only if we have data for both dates + if has_current_data and has_previous_data: + change = current_count - previous_count + print(f" Change: {change} (current - previous)") + if change > 0: + message += f"đ´ +{change} muted tests in last {period_days} days\n\n" + elif change < 0: + message += f"đĸ {change} muted tests in last {period_days} days\n\n" + else: + message += f"âĒ No change in last {period_days} days\n\n" + else: + print(f" Cannot calculate change: missing data for current={has_current_data}, previous={has_previous_data}") + + # Try to find the earliest available data for comparison + if trend_data and has_current_data: + available_dates = sorted(trend_data.keys()) + if len(available_dates) > 1: + earliest_date = available_dates[0] + earliest_count = trend_data[earliest_date] + days_span = (datetime.strptime(current_date_str, '%Y-%m-%d') - datetime.strptime(earliest_date, '%Y-%m-%d')).days + change = current_count - earliest_count + print(f" Using earliest available data: {earliest_date} ({earliest_count}) vs {current_date_str} ({current_count}) = {change} over {days_span} days") + + if change > 0: + message += f"đ´ +{change} muted tests since {earliest_date} ({days_span} days ago)\n\n" + elif change < 0: + message += f"đĸ {change} muted tests since {earliest_date} ({days_span} days ago)\n\n" + else: + message += f"âĒ No change since {earliest_date} ({days_span} days ago)\n\n" + else: + message += f"âšī¸ No historical data available for comparison\n\n" + else: + if not has_previous_data: + message += f"âšī¸ No data available for comparison ({period_days} days ago)\n\n" + else: + message += f"âšī¸ Insufficient data for trend analysis\n\n" + + # Add responsible users if available + if team_responsible and team_name in team_responsible: + responsible = team_responsible[team_name] + # Handle both single responsible and list of responsibles + if isinstance(responsible, list): + responsible_str = " ".join(f"@{r.replace('_', '\\_')}" if not r.startswith('@') else r.replace('_', '\\_') for r in responsible) + else: + responsible_str = f"@{responsible.replace('_', '\\_')}" if not responsible.startswith('@') else responsible.replace('_', '\\_') + message += f"fyi: {responsible_str}\n\n" + + # Escape the entire message for MarkdownV2 + message = escape_markdown(message) + + # Print final message before sending + print(f"đ Final {period}ly message for {team_name}:") + print("-" * 80) + print(message) + print("-" * 80) + + if dry_run: + print(f"đ [DRY RUN] Team: {team_name}") + print(f"đ Channel: {team_channel_name} ({team_chat_id})") + print(f"đ Thread: {team_thread_id}") + print("đ Message:") + print("-" * 80) + print(message) + print("-" * 80) + print() + success_count += 1 + else: + print(f"đ¨ Sending {period}ly update for team: {team_name}") + + # Get trend data for this team + trend_data = all_team_data.get(team_name, {}).get('trend', {}) + + if trend_data and MATPLOTLIB_AVAILABLE: + # Create trend plot + plot_data = create_trend_plot(team_name, trend_data, debug_plots_dir) + + if plot_data: + # Send message with plot + with tempfile.NamedTemporaryFile(suffix='.png', delete=False) as tmp_file: + tmp_file.write(base64.b64decode(plot_data)) + tmp_path = tmp_file.name + + try: + print(f"đ¤ Sending trend plot for team: {team_name}") + if send_telegram_message(bot_token, team_chat_id, message, "MarkdownV2", team_thread_id, True, max_retries, retry_delay, photo_path=tmp_path): + success_count += 1 + print(f"â
Trend update sent for team: {team_name}") + else: + print(f"â Failed to send trend update for team: {team_name}") + finally: + # Clean up temporary file + if os.path.exists(tmp_path): + os.unlink(tmp_path) + else: + # Fallback to text message + print(f"â ī¸ Could not create plot, sending text message for team: {team_name}") + if send_telegram_message(bot_token, team_chat_id, message, "MarkdownV2", team_thread_id, True, max_retries, retry_delay): + success_count += 1 + print(f"â
Text update sent for team: {team_name}") + else: + print(f"â Failed to send text update for team: {team_name}") + else: + # Send text message only + print(f"đ¤ Sending text update for team: {team_name}") + if send_telegram_message(bot_token, team_chat_id, message, "MarkdownV2", team_thread_id, True, max_retries, retry_delay): + success_count += 1 + print(f"â
Text update sent for team: {team_name}") + else: + print(f"â Failed to send text update for team: {team_name}") + + # Add delay between messages + if success_count < total_teams: + time.sleep(delay) + + if dry_run: + print(f"đ Dry run completed: {success_count}/{total_teams} {period}ly updates formatted!") + if blacklisted_count > 0: + print(f"đ Summary: {len(teams_from_data)} total teams, {blacklisted_count} blacklisted, {total_teams} processed") + else: + print(f"đ Sent {success_count}/{total_teams} {period}ly updates successfully!") + if blacklisted_count > 0: + print(f"đ Summary: {len(teams_from_data)} total teams, {blacklisted_count} blacklisted, {total_teams} processed, {success_count} sent") + + return success_count == total_teams + + def main(): parser = argparse.ArgumentParser(description="Parse team issues and send separate messages for each team") # Required arguments - parser.add_argument('--file', required=True, help='Path to file with formatted results') + parser.add_argument('--file', help='Path to file with formatted results (required for --on-mute-change-update mode)') parser.add_argument('--bot-token', help='Telegram bot token (or use TELEGRAM_BOT_TOKEN env var)') parser.add_argument('--team-channels', required=True, help='JSON string mapping teams to their channel configurations (or use TEAM_CHANNELS env var)') @@ -388,8 +1218,27 @@ def main(): parser.add_argument('--max-retries', type=int, default=5, help='Maximum number of retry attempts for failed messages (default: 5)') parser.add_argument('--retry-delay', type=int, default=10, help='Delay between retry attempts in seconds (default: 10)') + # Mode selection (mutually exclusive) + mode_group = parser.add_mutually_exclusive_group(required=True) + mode_group.add_argument('--on-mute-change-update', action='store_true', help='Default mode: send updates about new muted tests') + mode_group.add_argument('--period-update', choices=['week', 'month'], help='Send periodic trend updates (week or month)') + + # YDB arguments for muted tests statistics + parser.add_argument('--ydb-endpoint', help='YDB database endpoint (or use YDB_ENDPOINT env var)') + parser.add_argument('--ydb-database', help='YDB database path (or use YDB_DATABASE env var)') + parser.add_argument('--ydb-credentials', help='Path to YDB service account credentials JSON file (or use YDB_SERVICE_ACCOUNT_KEY_FILE_CREDENTIALS env var)') + parser.add_argument('--no-stats', action='store_true', help='Skip fetching muted tests statistics from YDB') + parser.add_argument('--use-yesterday', action='store_true', help='Use yesterday\'s data for development convenience') + parser.add_argument('--include-plots', action='store_true', help='Include trend plots in messages (requires matplotlib)') + parser.add_argument('--debug-plots-dir', help='Directory to save debug plot files (enables debug mode)') + args = parser.parse_args() + # Validate mode-specific requirements + if args.on_mute_change_update and not args.file: + print("â --file is required for --on-mute-change-update mode") + sys.exit(1) + # Get bot token bot_token = args.bot_token or os.getenv('TELEGRAM_BOT_TOKEN') @@ -405,6 +1254,60 @@ def main(): print(f"đ Loaded channel configurations for {len(team_channels.get('teams', {}))} teams") + # Handle period update mode + if args.period_update: + # Prepare YDB config for period updates + ydb_config = { + 'endpoint': args.ydb_endpoint, + 'path': args.ydb_database, + 'credentials': args.ydb_credentials, + 'use_yesterday': args.use_yesterday + } + + # Check if we need Telegram connection (not for dry run) + if not args.dry_run: + if not bot_token: + print("â Bot token not provided. Use --bot-token or set TELEGRAM_BOT_TOKEN environment variable") + sys.exit(1) + + # Send period updates + success = send_period_updates( + period=args.period_update, + bot_token=bot_token, + team_channels=team_channels, + ydb_config=ydb_config, + delay=args.delay, + max_retries=args.max_retries, + retry_delay=args.retry_delay, + dry_run=args.dry_run, + debug_plots_dir=args.debug_plots_dir + ) + + if success: + print("â
Period updates completed successfully") + sys.exit(0) + else: + print("â Period updates failed") + sys.exit(1) + + # Handle on-mute-change-update mode (default mode) + # Get muted tests statistics if not disabled + muted_stats = None + if not args.no_stats: + print("đ Fetching muted tests statistics from YDB...") + muted_stats = get_muted_tests_stats( + database_endpoint=args.ydb_endpoint, + database_path=args.ydb_database, + credentials_path=args.ydb_credentials, + use_yesterday=args.use_yesterday + ) + if muted_stats: + print(f"â
Statistics loaded for {len(muted_stats)} teams") + else: + print("â ī¸ Could not load statistics, continuing without stats") + else: + print("âī¸ Skipping statistics fetch (--no-stats flag)") + # Check if we need Telegram connection (not for dry run) if not args.dry_run or args.test_connection: if not bot_token: @@ -479,8 +1382,48 @@ def main(): print(f" - {team_name}: {len(issues)} issues{responsible_info}{channel_info}") + # Prepare YDB config and get all team data if needed + ydb_config = None + all_team_data = None + + if args.include_plots and not args.no_stats: + ydb_config = { + 'endpoint': args.ydb_endpoint, + 'path': args.ydb_database, + 'credentials': args.ydb_credentials, + 'use_yesterday': args.use_yesterday + } + + # Get all team data in one optimized query + print("đ Fetching all team data in one optimized query...") + all_team_data = get_all_team_data( + database_endpoint=args.ydb_endpoint, + database_path=args.ydb_database, + credentials_path=args.ydb_credentials, + use_yesterday=args.use_yesterday + ) + + if all_team_data: + print(f"â
Successfully fetched data for {len(all_team_data)} teams") + else: + print("â ī¸ Could not fetch team data, will use individual queries if needed") + # Send messages (or show in dry run) - send_team_messages(teams, bot_token, args.delay, args.max_retries, args.retry_delay, team_channels, args.dry_run) + send_team_messages( + teams, + bot_token, + args.delay, + args.max_retries, + args.retry_delay, + team_channels, + args.dry_run, + muted_stats, + args.include_plots, + ydb_config, + args.debug_plots_dir, + all_team_data, + MUTE_UPDATE_SHOW_DIFF + ) if __name__ == "__main__": diff --git a/.github/scripts/telegram/send_telegram_message.py b/.github/scripts/telegram/send_telegram_message.py index d16fd348ca0..5ac64808408 100644 --- a/.github/scripts/telegram/send_telegram_message.py +++ b/.github/scripts/telegram/send_telegram_message.py @@ -12,50 +12,50 @@ import time from pathlib import Path -def send_telegram_message(bot_token, chat_id, message_or_file, parse_mode="Markdown", message_thread_id=None, disable_web_page_preview=True, max_retries=5, retry_delay=10, delay=1): +def _read_content(message_or_file): """ - Send a message to Telegram channel with retry mechanism. - Can accept either text message or file path. + Read content from message string or file. + + Args: + message_or_file (str): Message text or path to file to send + + Returns: + str: Content to send, or None if error + """ + # Try to open as file first, if it fails, treat as message + try: + with open(message_or_file, 'r', encoding='utf-8') as f: + return f.read() + except (OSError, IOError, ValueError, UnicodeDecodeError): + # It's a text message, not a file (or file can't be read as text) + return message_or_file + + +def _send_chunked_messages(bot_token, chat_id, content, parse_mode, message_thread_id, disable_web_page_preview, max_retries, retry_delay, delay, max_length=4000): + """ + Send chunked text messages. Args: bot_token (str): Telegram bot token chat_id (str): Telegram chat ID - message_or_file (str): Message text or path to file to send + content (str): Content to send parse_mode (str): Parse mode for message formatting message_thread_id (int, optional): Thread ID for group messages disable_web_page_preview (bool): Disable web page preview for links max_retries (int): Maximum number of retry attempts retry_delay (int): Delay between retry attempts in seconds delay (int): Delay between message chunks in seconds + max_length (int): Maximum length per chunk Returns: - bool: True if successful, False otherwise + bool: True if all chunks sent successfully, False otherwise """ - # Check if message_or_file is a file path - # If the string is too long, it's definitely not a file path - if len(message_or_file) > 255: - # It's definitely a text message, not a file path - content = message_or_file - else: - file_path = Path(message_or_file) - if file_path.exists() and file_path.is_file(): - # It's a file, read its content - try: - with open(file_path, 'r', encoding='utf-8') as f: - content = f.read() - except Exception as e: - print(f"â Error reading file {file_path}: {e}") - return False - else: - # It's a text message - content = message_or_file - if not content.strip(): print(f"â ī¸ Content is empty") return True # Split message into chunks if needed - chunks = split_message(content) + chunks = split_message(content, max_length) print(f"đ¤ Sending {len(chunks)} message(s)...") success_count = 0 @@ -77,6 +77,104 @@ def send_telegram_message(bot_token, chat_id, message_or_file, parse_mode="Markd return False +def _send_photo_with_chunked_caption(bot_token, chat_id, photo_path, content, parse_mode, message_thread_id, disable_web_page_preview, max_retries, retry_delay, delay): + """ + Send photo with chunked caption. + + Args: + bot_token (str): Telegram bot token + chat_id (str): Telegram chat ID + photo_path (str): Path to photo file + content (str): Caption content + parse_mode (str): Parse mode for message formatting + message_thread_id (int, optional): Thread ID for group messages + disable_web_page_preview (bool): Disable web page preview for links + max_retries (int): Maximum number of retry attempts + retry_delay (int): Delay between retry attempts in seconds + delay (int): Delay between message chunks in seconds + + Returns: + bool: True if all chunks sent successfully, False otherwise + """ + photo_file = Path(photo_path) + if not photo_file.exists() or not photo_file.is_file(): + print(f"â Photo file not found: {photo_path}") + return False + + print(f"đˇ Sending photo: {photo_path}") + + # Split message into chunks for photo caption + chunks = split_message(content, max_length=1024) # Telegram caption limit is 1024 characters + print(f"đ¤ Sending photo with {len(chunks)} message chunk(s)...") + + success_count = 0 + for i, chunk in enumerate(chunks, 1): + print(f"đ¨ Sending photo chunk {i}/{len(chunks)}...") + + if i == 1: + # First chunk goes with photo + if _send_photo(bot_token, chat_id, photo_path, chunk, parse_mode, message_thread_id, max_retries, retry_delay): + success_count += 1 + print(f"â
Photo with caption sent successfully!") + else: + print(f"â Failed to send photo with caption") + else: + # Subsequent chunks as regular messages + if _send_single_message(bot_token, chat_id, chunk, parse_mode, message_thread_id, disable_web_page_preview, max_retries, retry_delay): + success_count += 1 + print(f"â
Text chunk {i} sent successfully!") + else: + print(f"â Failed to send text chunk {i}") + + # Add delay between messages (except for the last one) + if i < len(chunks): + time.sleep(delay) + + if success_count == len(chunks): + print(f"â
All {success_count} message(s) sent successfully!") + return True + else: + print(f"â ī¸ Only {success_count}/{len(chunks)} message(s) sent successfully") + return False + + +def send_telegram_message(bot_token, chat_id, message_or_file, parse_mode="MarkdownV2", message_thread_id=None, disable_web_page_preview=True, max_retries=5, retry_delay=10, delay=1, photo_path=None): + """ + Send a message to Telegram channel with retry mechanism. + Can accept either text message or file path. + + Args: + bot_token (str): Telegram bot token + chat_id (str): Telegram chat ID + message_or_file (str): Message text or path to file to send + parse_mode (str): Parse mode for message formatting + message_thread_id (int, optional): Thread ID for group messages + disable_web_page_preview (bool): Disable web page preview for links + max_retries (int): Maximum number of retry attempts + retry_delay (int): Delay between retry attempts in seconds + delay (int): Delay between message chunks in seconds + photo_path (str, optional): Path to photo file to send + + Returns: + bool: True if successful, False otherwise + """ + print(f"đ send_telegram_message called with photo_path: {photo_path}") + print(f"đ message_or_file length: {len(message_or_file) if message_or_file else 'None'}") + print(f"đ chat_id: {chat_id}, message_thread_id: {message_thread_id}") + + # Read content from message string or file + content = _read_content(message_or_file) + if content is None: + return False + + # If photo is provided, send photo with chunked caption + if photo_path: + return _send_photo_with_chunked_caption(bot_token, chat_id, photo_path, content, parse_mode, message_thread_id, disable_web_page_preview, max_retries, retry_delay, delay) + + # If no photo, send text message + return _send_chunked_messages(bot_token, chat_id, content, parse_mode, message_thread_id, disable_web_page_preview, max_retries, retry_delay, delay) + + def _send_single_message(bot_token, chat_id, message, parse_mode, message_thread_id, disable_web_page_preview, max_retries, retry_delay): """ Send a single message to Telegram channel with retry mechanism. @@ -127,13 +225,14 @@ def _send_single_message(bot_token, chat_id, message, parse_mode, message_thread return False result = response.json() + print(f"đ Full Telegram API Response: {result}") + if result.get('ok'): thread_info = f" (thread {message_thread_id})" if message_thread_id is not None else "" print(f"â
Message sent successfully to chat {chat_id}{thread_info}") return True else: print(f"â Telegram API Error: {result.get('description', 'Unknown error')}") - print(f"â Full response: {result}") if attempt < max_retries: print(f"âŗ Waiting {retry_delay} seconds before retry...") continue @@ -163,6 +262,96 @@ def _send_single_message(bot_token, chat_id, message, parse_mode, message_thread return False +def _send_photo(bot_token, chat_id, photo_path, caption, parse_mode, message_thread_id, max_retries, retry_delay): + """ + Send a photo to Telegram channel with retry mechanism. + + Args: + bot_token (str): Telegram bot token + chat_id (str): Telegram chat ID + photo_path (str): Path to photo file + caption (str): Photo caption + parse_mode (str): Parse mode for caption formatting + message_thread_id (int, optional): Thread ID for group messages + max_retries (int): Maximum number of retry attempts + retry_delay (int): Delay between retry attempts in seconds + + Returns: + bool: True if successful, False otherwise + """ + for attempt in range(max_retries + 1): + if attempt > 0: + print(f"đ Retry attempt {attempt}/{max_retries}...") + time.sleep(retry_delay) + + url = f"https://api.telegram.org/bot{bot_token}/sendPhoto" + + # Prepare files and data for multipart/form-data + with open(photo_path, 'rb') as photo_file: + files = {'photo': photo_file} + + data = { + 'chat_id': chat_id, + 'caption': caption, + 'parse_mode': parse_mode + } + + # Add thread ID if provided + if message_thread_id is not None: + data['message_thread_id'] = message_thread_id + + try: + response = requests.post(url, files=files, data=data, timeout=60) + + # Always print response for debugging + print(f"đ Telegram API Response: {response.status_code}") + + if response.status_code != 200: + print(f"â HTTP Error {response.status_code}: {response.text}") + if attempt < max_retries: + print(f"âŗ Waiting {retry_delay} seconds before retry...") + continue + else: + return False + + result = response.json() + print(f"đ Full Telegram API Response: {result}") + + if result.get('ok'): + thread_info = f" (thread {message_thread_id})" if message_thread_id is not None else "" + print(f"â
Photo sent successfully to chat {chat_id}{thread_info}") + return True + else: + print(f"â Telegram API Error: {result.get('description', 'Unknown error')}") + if attempt < max_retries: + print(f"âŗ Waiting {retry_delay} seconds before retry...") + continue + else: + return False + + except requests.exceptions.RequestException as e: + print(f"â Network error: {e}") + if attempt < max_retries: + print(f"âŗ Waiting {retry_delay} seconds before retry...") + continue + else: + return False + except Exception as e: + print(f"â Unexpected error: {e}") + if attempt < max_retries: + print(f"âŗ Waiting {retry_delay} seconds before retry...") + continue + else: + return False + + # If all retries failed, print the caption that couldn't be sent + print(f"â Failed to send photo after {max_retries} retries. Caption content:") + print("=" * 80) + print(caption) + print("=" * 80) + return False + + def split_message(message, max_length=4000): """ Split long message into chunks. @@ -211,8 +400,8 @@ def main(): parser.add_argument('--message', required=True, help='Message text or path to file to send') # Optional arguments - parser.add_argument('--parse-mode', default='Markdown', choices=['Markdown', 'HTML', 'None'], - help='Parse mode for message formatting (default: Markdown)') + parser.add_argument('--parse-mode', default='MarkdownV2', choices=['Markdown', 'MarkdownV2', 'HTML', 'None'], + help='Parse mode for message formatting (default: MarkdownV2)') parser.add_argument('--delay', type=int, default=1, help='Delay between message chunks in seconds (default: 1)') parser.add_argument('--max-length', type=int, default=4000, @@ -225,6 +414,8 @@ def main(): help='Maximum number of retry attempts for failed messages (default: 5)') parser.add_argument('--retry-delay', type=int, default=10, help='Delay between retry attempts in seconds (default: 10)') + parser.add_argument('--photo', + help='Path to photo file to send (optional)') args = parser.parse_args() @@ -250,7 +441,8 @@ def main(): disable_web_page_preview=args.disable_web_page_preview, max_retries=args.max_retries, retry_delay=args.retry_delay, - delay=args.delay + delay=args.delay, + photo_path=args.photo ) sys.exit(0 if success else 1) diff --git a/.github/workflows/create_issues_for_muted_tests.yml b/.github/workflows/create_issues_for_muted_tests.yml index 192593d9910..35ec820147e 100644 --- a/.github/workflows/create_issues_for_muted_tests.yml +++ b/.github/workflows/create_issues_for_muted_tests.yml @@ -41,7 +41,7 @@ jobs: - name: Install dependencies run: | python -m pip install --upgrade pip - pip install ydb[yc] PyGithub requests + pip install ydb[yc] PyGithub requests matplotlib numpy - name: Setup ydb access uses: ./.github/actions/setup_ci_ydb_service_account_key_file_credentials @@ -85,7 +85,9 @@ jobs: TEAM_CHANNELS: ${{ vars.TG_TEAM_CHANNELS }} run: | python .github/scripts/telegram/parse_and_send_team_issues.py \ + --on-mute-change-update \ --file "${{ steps.create_issues.outputs.created_issues_file }}" \ --bot-token "$TELEGRAM_BOT_TOKEN" \ --team-channels "$TEAM_CHANNELS" \ + --include-plots \ --delay 2 diff --git a/.github/workflows/weekly_telegram_update.yml b/.github/workflows/weekly_telegram_update.yml new file mode 100644 index 00000000000..9195f53a012 --- /dev/null +++ b/.github/workflows/weekly_telegram_update.yml @@ -0,0 +1,46 @@ +name: Weekly Telegram Update + +on: + schedule: + # Every Monday at 11:00 MSK (08:00 UTC) + - cron: "0 8 * * 1" + workflow_dispatch: + +defaults: + run: + shell: bash + +env: + GH_TOKEN: ${{ secrets.YDBOT_TOKEN }} + +jobs: + weekly-telegram-update: + name: Send weekly trend updates to Telegram + runs-on: [ self-hosted, auto-provisioned, build-preset-analytic-node] + + steps: + - name: Checkout + uses: actions/checkout@v4 + with: + ref: main + + - name: Setup ydb access + uses: ./.github/actions/setup_ci_ydb_service_account_key_file_credentials + with: + ci_ydb_service_account_key_file_credentials: ${{ secrets.CI_YDB_SERVICE_ACCOUNT_KEY_FILE_CREDENTIALS }} + + - name: Install dependencies + run: | + python3 -m pip install --upgrade pip + pip install ydb[yc] requests matplotlib numpy + + - name: Send weekly trend updates to Telegram + env: + TELEGRAM_BOT_TOKEN: ${{ secrets.TELEGRAM_YDBOT_TOKEN }} + TEAM_CHANNELS: ${{ vars.TG_TEAM_CHANNELS }} + run: | + python3 .github/scripts/telegram/parse_and_send_team_issues.py \ + --period-update week \ + --bot-token "$TELEGRAM_BOT_TOKEN" \ + --team-channels "$TEAM_CHANNELS" \ + --delay 2 |
