summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorKirill Rysin <[email protected]>2025-09-28 04:32:57 +0200
committerGitHub <[email protected]>2025-09-28 05:32:57 +0300
commitdb154818a7ba839a7dc7b99f28fa5baec43e00ec (patch)
tree12ec3296da729b71cb583bd038537d3c72458af2
parentce007905a0d0675a73d4aee3d5f7c5963923f1c0 (diff)
Mute bot: statistic in telegram (#25846)
-rw-r--r--.github/scripts/telegram/README.md10
-rw-r--r--.github/scripts/telegram/README_parse_and_send.md261
-rw-r--r--.github/scripts/telegram/README_team_issues.md9
-rw-r--r--.github/scripts/telegram/README_telegram.md9
-rw-r--r--.github/scripts/telegram/parse_and_send_team_issues.py983
-rw-r--r--.github/scripts/telegram/send_telegram_message.py250
-rw-r--r--.github/workflows/create_issues_for_muted_tests.yml4
-rw-r--r--.github/workflows/weekly_telegram_update.yml46
8 files changed, 1514 insertions, 58 deletions
diff --git a/.github/scripts/telegram/README.md b/.github/scripts/telegram/README.md
index eec4a5c15bd..8e376e1a166 100644
--- a/.github/scripts/telegram/README.md
+++ b/.github/scripts/telegram/README.md
@@ -17,17 +17,19 @@ General-purpose script for sending messages or file contents to Telegram channel
**Documentation:** [README_telegram.md](README_telegram.md)
### đŸ‘Ĩ `parse_and_send_team_issues.py`
-Specialized script for parsing GitHub issues and sending team-specific notifications.
+Specialized script for parsing GitHub issues and sending team-specific notifications. Supports two modes: immediate updates and periodic trend reports.
**Features:**
-- Parse team issues from formatted results
-- Send separate messages for each team
+- **On-Mute-Change Updates**: Parse team issues from formatted results and send immediate notifications
+- **Periodic Trend Updates**: Send weekly/monthly trend reports with statistics and charts
- Support for multiple responsible users per team
- Team lead mentions in messages
- Support for message threads
- Dry run mode for testing
+- Automatic team discovery from YDB data
+- Fallback to default channel for teams without specific configuration
-**Documentation:** [README_team_issues.md](README_team_issues.md)
+**Documentation:** [README_parse_and_send.md](README_parse_and_send.md)
## Configuration
diff --git a/.github/scripts/telegram/README_parse_and_send.md b/.github/scripts/telegram/README_parse_and_send.md
new file mode 100644
index 00000000000..3a9b6981519
--- /dev/null
+++ b/.github/scripts/telegram/README_parse_and_send.md
@@ -0,0 +1,261 @@
+# Parse and Send Team Issues
+
+Script to parse GitHub issues results and send separate messages for each team with muted tests statistics. Supports two modes: immediate updates and periodic trend reports.
+
+## Quick Start
+
+### Mode 1: On-Mute-Change Updates (Default)
+Send immediate notifications about new muted tests:
+
+```bash
+# Basic usage with YDB statistics
+python .github/scripts/telegram/parse_and_send_team_issues.py \
+ --on-mute-change-update \
+ --file "formatted_results.txt" \
+ --team-channels '{"teams": {"team-name": {"responsible": ["@user1", "@user2"], "channel": "channel-name"}}, "channels": {"channel-name": "123456789/1"}}' \
+ --bot-token "YOUR_BOT_TOKEN"
+
+# With custom YDB settings
+python .github/scripts/telegram/parse_and_send_team_issues.py \
+ --on-mute-change-update \
+ --file "formatted_results.txt" \
+ --team-channels '{"teams": {"team-name": {"responsible": ["@user1"], "channel": "channel-name"}}, "channels": {"channel-name": "123456789/1"}}' \
+ --bot-token "YOUR_BOT_TOKEN" \
+ --ydb-endpoint "grpcs://ydb.serverless.yandexcloud.net:2135" \
+ --ydb-database "/ru-central1/b1g8ejbrie0sfh5k0j2j/etn8l4e3hbti8k4n5g2e" \
+ --ydb-credentials "path/to/credentials.json"
+
+# Dry run (show messages without sending)
+python .github/scripts/telegram/parse_and_send_team_issues.py \
+ --on-mute-change-update \
+ --file "formatted_results.txt" \
+ --team-channels '{"teams": {"team-name": {"responsible": ["@user1"], "channel": "channel-name"}}, "channels": {"channel-name": "123456789/1"}}' \
+ --dry-run
+
+# Skip statistics fetch
+python .github/scripts/telegram/parse_and_send_team_issues.py \
+ --on-mute-change-update \
+ --file "formatted_results.txt" \
+ --team-channels '{"teams": {"team-name": {"responsible": ["@user1"], "channel": "channel-name"}}, "channels": {"channel-name": "123456789/1"}}' \
+ --bot-token "YOUR_BOT_TOKEN" \
+ --no-stats
+```
+
+### Mode 2: Periodic Trend Updates
+Send weekly or monthly trend reports with statistics and charts:
+
+```bash
+# Weekly trend updates
+python .github/scripts/telegram/parse_and_send_team_issues.py \
+ --period-update week \
+ --team-channels '{"default_channel": "main_channel", "teams": {"team-name": {"responsible": ["@user1"], "channel": "main_channel"}}, "channels": {"main_channel": "123456789/1"}}' \
+ --bot-token "YOUR_BOT_TOKEN" \
+ --ydb-endpoint "grpcs://ydb.serverless.yandexcloud.net:2135" \
+ --ydb-database "/ru-central1/b1g8ejbrie0sfh5k0j2j/etn8l4e3hbti8k4n5g2e"
+
+# Monthly trend updates with debug plots
+python .github/scripts/telegram/parse_and_send_team_issues.py \
+ --period-update month \
+ --team-channels '{"default_channel": "main_channel", "teams": {"team-name": {"responsible": ["@user1"], "channel": "main_channel"}}, "channels": {"main_channel": "123456789/1"}}' \
+ --bot-token "YOUR_BOT_TOKEN" \
+ --ydb-endpoint "grpcs://ydb.serverless.yandexcloud.net:2135" \
+ --ydb-database "/ru-central1/b1g8ejbrie0sfh5k0j2j/etn8l4e3hbti8k4n5g2e" \
+ --debug-plots-dir "/path/to/debug/plots" \
+ --dry-run
+```
+
+## Parameters
+
+### Mode Selection (Required - Choose One)
+- `--on-mute-change-update` - Default mode: send updates about new muted tests (requires --file)
+- `--period-update {week,month}` - Send periodic trend updates (no --file required)
+
+### Required
+- `--team-channels` - JSON string mapping teams to their channel configurations (or use TEAM_CHANNELS env var)
+- `--file` - Path to file with formatted results (required only for --on-mute-change-update mode)
+
+### Optional
+- `--bot-token` - Telegram bot token (or use TELEGRAM_BOT_TOKEN env var)
+- `--delay` - Delay between messages in seconds (default: 2)
+- `--dry-run` - Parse and show teams without sending messages
+- `--test-connection` - Test Telegram connection only
+- `--message-thread-id` - Thread ID for group messages (optional)
+- `--max-retries` - Maximum number of retry attempts for failed messages (default: 5)
+- `--retry-delay` - Delay between retry attempts in seconds (default: 10)
+
+### YDB Statistics
+- `--ydb-endpoint` - YDB database endpoint (or use YDB_ENDPOINT env var)
+- `--ydb-database` - YDB database path (or use YDB_DATABASE env var)
+- `--ydb-credentials` - Path to YDB service account credentials JSON file (or use YDB_SERVICE_ACCOUNT_KEY_FILE_CREDENTIALS env var)
+- `--no-stats` - Skip fetching muted tests statistics from YDB (only for --on-mute-change-update)
+- `--include-plots` - Include trend plots in messages (requires matplotlib)
+- `--debug-plots-dir` - Directory to save debug plot files (enables debug mode)
+- `--use-yesterday` - Use yesterday's data for development convenience
+
+## Features
+
+### On-Mute-Change Updates Mode
+- ✅ Automatic parsing of team issues from formatted results
+- ✅ Separate messages for each team
+- ✅ Muted tests statistics from YDB (total, today, and minus today counts)
+- ✅ Monthly trend plots with matplotlib
+- ✅ Team-specific channel routing
+- ✅ Responsible user mentions
+- ✅ Markdown formatting support
+- ✅ Delay between messages to avoid API limits
+- ✅ Message thread support
+- ✅ Automatic retry mechanism
+- ✅ Dry run mode for testing
+- ✅ Connection testing
+
+### Periodic Trend Updates Mode
+- ✅ Weekly and monthly trend reports
+- ✅ Automatic team discovery from YDB data
+- ✅ Team blacklist support (exclude specific teams from periodic updates)
+- ✅ Fallback to default channel for teams without specific configuration
+- ✅ Trend charts with 30-day history
+- ✅ Period-over-period change calculations
+- ✅ Color-coded statistics (red for increases, green for decreases)
+- ✅ Dashboard links for each team
+- ✅ Responsible user mentions (if team configured)
+- ✅ Team hashtags for easy filtering
+
+## Message Format
+
+### On-Mute-Change Updates Mode
+The script sends messages in the following format:
+
+```
+🔇 **13-01-25 new muted tests in `main` for [team-name](https://github.com/orgs/ydb-platform/teams/team-name)** #teamname
+
+ đŸŽ¯ `Issue Title` [#12345](https://github.com/...)
+ đŸŽ¯ `Another Issue Title` [#12346](https://github.com/...)
+
+📊 **[Total muted tests: 150](https://datalens.yandex/4un3zdm0zcnyr?owner_team=team-name) 🔴+5 muted /đŸŸĸ-2 unmuted**
+
+fyi: @user1 @user2
+```
+
+**Statistics explanation:**
+- `📊 **[Total muted tests: N](dashboard_url) 🔴+M muted /đŸŸĸ-K unmuted**` - Total muted tests with today's changes
+- `📊 **[Total muted tests: N](dashboard_url) 🔴+M muted**` - Total muted tests with today's additions only
+- `📊 **[Total muted tests: N](dashboard_url) đŸŸĸ-K unmuted**` - Total muted tests with today's unmutes only
+- `📊 **[Total muted tests: N](dashboard_url)**` - Total muted tests (no changes today)
+
+### Periodic Trend Updates Mode
+The script sends trend reports in the following format:
+
+```
+📈 **Week Over Week changes for team [team-name](https://github.com/orgs/ydb-platform/teams/team-name)** #teamname
+
+📊 **[Total muted tests: 150](https://datalens.yandex/4un3zdm0zcnyr?owner_team=team-name) (🔴+10 vs 7 days ago)**
+
+fyi: @user1 @user2
+
+Chart shows muted tests trend over the last 30 days.
+```
+
+**Period statistics explanation:**
+- `(🔴+N vs X days ago)` - Increase compared to previous period
+- `(đŸŸĸ-N vs X days ago)` - Decrease compared to previous period
+- `(0 vs X days ago)` - No change compared to previous period
+
+## Team Channels Configuration
+
+The `--team-channels` parameter expects a JSON configuration:
+
+```json
+{
+ "default_channel": "default-channel-name",
+ "teams": {
+ "team-name": {
+ "responsible": ["@user1", "@user2"],
+ "channel": "specific-channel-name"
+ },
+ "another-team": {
+ "responsible": "@single-user",
+ "channel": "another-channel"
+ }
+ },
+ "channels": {
+ "default-channel-name": "123456789/1",
+ "specific-channel-name": "987654321/2",
+ "another-channel": "555666777"
+ }
+}
+```
+
+## Environment Variables
+
+- `TELEGRAM_BOT_TOKEN` - Telegram bot token
+- `TEAM_CHANNELS` - Team channels configuration JSON
+- `YDB_ENDPOINT` - YDB database endpoint
+- `YDB_DATABASE` - YDB database path
+- `YDB_SERVICE_ACCOUNT_KEY_FILE_CREDENTIALS` - Path to YDB credentials file
+
+## Team Blacklist
+
+The script supports a blacklist for periodic updates (weekly/monthly). Teams in the blacklist will be skipped during periodic trend updates but will still receive immediate mute-change notifications.
+
+To add teams to the blacklist, edit the `PERIOD_UPDATE_BLACKLIST` constant in the script:
+
+```python
+# Teams blacklisted from weekly/monthly updates
+PERIOD_UPDATE_BLACKLIST = {
+ 'storage', # Example: storage team
+ 'team-name' # Add more teams as needed
+}
+```
+
+**Note:** The blacklist only affects periodic updates (`--period-update` mode), not immediate mute-change notifications (`--on-mute-change-update` mode).
+
+## Dependencies
+
+- `ydb` - YDB Python client library
+- `requests` - HTTP library for Telegram API
+- `telegram` - Telegram bot functionality (from send_telegram_message.py)
+- `matplotlib` - For creating trend plots (optional, only if --include-plots is used)
+
+## Examples with Plots
+
+### On-Mute-Change Updates with Plots
+```bash
+# Send messages with trend plots
+python .github/scripts/telegram/parse_and_send_team_issues.py \
+ --on-mute-change-update \
+ --file "formatted_results.txt" \
+ --team-channels '{"teams": {"team-name": {"responsible": ["@user1"], "channel": "channel-name"}}, "channels": {"channel-name": "123456789/1"}}' \
+ --bot-token "YOUR_BOT_TOKEN" \
+ --include-plots \
+ --use-yesterday
+
+# Send messages with debug plots saved to custom directory
+python .github/scripts/telegram/parse_and_send_team_issues.py \
+ --on-mute-change-update \
+ --file "formatted_results.txt" \
+ --team-channels '{"teams": {"team-name": {"responsible": ["@user1"], "channel": "channel-name"}}, "channels": {"channel-name": "123456789/1"}}' \
+ --bot-token "YOUR_BOT_TOKEN" \
+ --include-plots \
+ --debug-plots-dir "/path/to/debug/plots"
+```
+
+### Periodic Trend Updates (Always Include Plots)
+```bash
+# Weekly trend updates with debug plots
+python .github/scripts/telegram/parse_and_send_team_issues.py \
+ --period-update week \
+ --team-channels '{"default_channel": "main_channel", "teams": {"team-name": {"responsible": ["@user1"], "channel": "main_channel"}}, "channels": {"main_channel": "123456789/1"}}' \
+ --bot-token "YOUR_BOT_TOKEN" \
+ --ydb-endpoint "grpcs://ydb.serverless.yandexcloud.net:2135" \
+ --ydb-database "/ru-central1/b1g8ejbrie0sfh5k0j2j/etn8l4e3hbti8k4n5g2e" \
+ --debug-plots-dir "/path/to/debug/plots"
+
+# Monthly trend updates (dry run)
+python .github/scripts/telegram/parse_and_send_team_issues.py \
+ --period-update month \
+ --team-channels '{"default_channel": "main_channel", "teams": {"team-name": {"responsible": ["@user1"], "channel": "main_channel"}}, "channels": {"main_channel": "123456789/1"}}' \
+ --bot-token "YOUR_BOT_TOKEN" \
+ --ydb-endpoint "grpcs://ydb.serverless.yandexcloud.net:2135" \
+ --ydb-database "/ru-central1/b1g8ejbrie0sfh5k0j2j/etn8l4e3hbti8k4n5g2e" \
+ --dry-run
+```
diff --git a/.github/scripts/telegram/README_team_issues.md b/.github/scripts/telegram/README_team_issues.md
index 363d59e9819..20fc0d5e743 100644
--- a/.github/scripts/telegram/README_team_issues.md
+++ b/.github/scripts/telegram/README_team_issues.md
@@ -25,13 +25,14 @@ python .github/scripts/telegram/parse_and_send_team_issues.py \
## Message Format
```
-🔇 **07-09-24 new muted tests for [team-name](https://github.com/orgs/ydb-platform/teams/team-name)** #team-name
+🔇 **07-09-24 new muted tests in `main` for [team-name](https://github.com/orgs/ydb-platform/teams/team-name)** #teamname
-fyi: @responsible1 @responsible2
+ đŸŽ¯ `Issue Title` [#12345](https://github.com/...)
+ đŸŽ¯ `Another Issue Title` [#12346](https://github.com/...)
- - đŸŽ¯ [Issue URL](Issue URL) - `Issue Title`
- - đŸŽ¯ [Issue URL](Issue URL) - `Issue Title`
+📊 **[Total muted tests: 150](https://datalens.yandex/4un3zdm0zcnyr?owner_team=team-name) 🔴+5 muted /đŸŸĸ-2 unmuted**
+fyi: @responsible1 @responsible2
```
## Team Channels Configuration
diff --git a/.github/scripts/telegram/README_telegram.md b/.github/scripts/telegram/README_telegram.md
index 89b5bbdd554..4512d514188 100644
--- a/.github/scripts/telegram/README_telegram.md
+++ b/.github/scripts/telegram/README_telegram.md
@@ -16,6 +16,13 @@ python .github/scripts/telegram/send_telegram_message.py \
--message "Hello, World!" \
--bot-token "YOUR_BOT_TOKEN" \
--chat-id "1234567890"
+
+# Send photo with caption
+python .github/scripts/telegram/send_telegram_message.py \
+ --message "Check out this image!" \
+ --photo "path/to/image.jpg" \
+ --bot-token "YOUR_BOT_TOKEN" \
+ --chat-id "1234567890"
```
## Parameters
@@ -23,6 +30,7 @@ python .github/scripts/telegram/send_telegram_message.py \
- `--bot-token` - Telegram bot token (required)
- `--chat-id` - Chat/channel ID (required)
- `--message` - Message text or path to file to send (automatically detected)
+- `--photo` - Path to photo file to send (optional)
- `--parse-mode` - Parse mode (Markdown, HTML, None) - default: Markdown
- `--delay` - Delay between messages in seconds - default: 1
- `--max-length` - Maximum message length - default: 4000
@@ -35,6 +43,7 @@ python .github/scripts/telegram/send_telegram_message.py \
- ✅ Automatic splitting of long messages into chunks
- ✅ Markdown formatting support
+- ✅ Photo sending with caption support
- ✅ Delay between messages to avoid API limits
- ✅ Message thread support
- ✅ Disable link previews
diff --git a/.github/scripts/telegram/parse_and_send_team_issues.py b/.github/scripts/telegram/parse_and_send_team_issues.py
index 0a3e68ff5e0..04a3047d9ee 100644
--- a/.github/scripts/telegram/parse_and_send_team_issues.py
+++ b/.github/scripts/telegram/parse_and_send_team_issues.py
@@ -10,10 +10,420 @@ import requests
import argparse
import re
import json
-from datetime import datetime
+import configparser
+import tempfile
+import base64
+import shutil
+from datetime import datetime, timedelta
from pathlib import Path
from send_telegram_message import send_telegram_message
+try:
+ import ydb
+ YDB_AVAILABLE = True
+except ImportError:
+ YDB_AVAILABLE = False
+ print("âš ī¸ YDB client not available. Install with: pip install ydb")
+
+try:
+ import matplotlib.pyplot as plt
+ import matplotlib.dates as mdates
+ import numpy as np
+ MATPLOTLIB_AVAILABLE = True
+except ImportError:
+ MATPLOTLIB_AVAILABLE = False
+ print("âš ī¸ Matplotlib not available. Install with: pip install matplotlib")
+
+# Configuration constants
+MUTE_UPDATE_SHOW_DIFF = False # Set to True to show +/- statistics in mute update messages
+
+# Teams blacklisted from weekly/monthly updates
+PERIOD_UPDATE_BLACKLIST = {
+ 'storage' # Add team names that should not receive periodic updates
+}
+
+# URL constants (only for duplicated URLs)
+GITHUB_ORG_TEAMS_URL = "https://github.com/orgs/ydb-platform/teams"
+DATALENS_DASHBOARD_URL = "https://datalens.yandex/4un3zdm0zcnyr?owner_team={team_name}"
+
+
+def _setup_ydb_connection(database_endpoint=None, database_path=None, credentials_path=None):
+ """
+ Set up YDB connection parameters and credentials.
+
+ Args:
+ database_endpoint (str): YDB database endpoint
+ database_path (str): YDB database path
+ credentials_path (str): Path to service account credentials JSON file
+
+ Returns:
+ tuple: (database_endpoint, database_path) or (None, None) if error
+ """
+ if not YDB_AVAILABLE:
+ print("❌ YDB client not available")
+ return None, None
+
+ try:
+ # Set up credentials if provided
+ if credentials_path and os.path.exists(credentials_path):
+ os.environ["YDB_SERVICE_ACCOUNT_KEY_FILE_CREDENTIALS"] = credentials_path
+ elif "CI_YDB_SERVICE_ACCOUNT_KEY_FILE_CREDENTIALS" in os.environ:
+ os.environ["YDB_SERVICE_ACCOUNT_KEY_FILE_CREDENTIALS"] = os.environ["CI_YDB_SERVICE_ACCOUNT_KEY_FILE_CREDENTIALS"]
+
+ # Try to load from config file if not provided
+ if not database_endpoint or not database_path:
+ try:
+ config = configparser.ConfigParser()
+ config_file_path = os.path.join(os.path.dirname(__file__), "../../config/ydb_qa_db.ini")
+ if os.path.exists(config_file_path):
+ config.read(config_file_path)
+ if not database_endpoint and "QA_DB" in config and "DATABASE_ENDPOINT" in config["QA_DB"]:
+ database_endpoint = config["QA_DB"]["DATABASE_ENDPOINT"]
+ if not database_path and "QA_DB" in config and "DATABASE_PATH" in config["QA_DB"]:
+ database_path = config["QA_DB"]["DATABASE_PATH"]
+ except Exception as e:
+ print(f"âš ī¸ Could not load config file: {e}")
+
+ # Use default values if still not provided
+ if not database_endpoint:
+ database_endpoint = os.getenv('YDB_ENDPOINT', 'grpcs://ydb.serverless.yandexcloud.net:2135')
+ if not database_path:
+ database_path = os.getenv('YDB_DATABASE', '/ru-central1/b1g8ejbrie0sfh5k0j2j/etn8l4e3hbti8k4n5g2e')
+
+ return database_endpoint, database_path
+
+ except Exception as e:
+ print(f"❌ Error setting up YDB connection: {e}")
+ return None, None
+
+
+def _execute_ydb_query(database_endpoint, database_path, query, description):
+ """
+ Execute a YDB query and return results.
+
+ Args:
+ database_endpoint (str): YDB database endpoint
+ database_path (str): YDB database path
+ query (str): SQL query to execute
+ description (str): Description for logging
+
+ Returns:
+ list: Query results or None if error
+ """
+ try:
+ print(f"🔍 {description}")
+
+ with ydb.Driver(
+ endpoint=database_endpoint,
+ database=database_path,
+ credentials=ydb.credentials_from_env_variables(),
+ ) as driver:
+ driver.wait(timeout=10, fail_fast=True)
+ print("✅ Successfully connected to YDB")
+
+ table_client = ydb.TableClient(driver, ydb.TableClientSettings())
+ query_obj = ydb.ScanQuery(query, {})
+ it = table_client.scan_query(query_obj)
+ results = []
+ for result in it:
+ results.extend(result.result_set.rows)
+
+ print(f"📊 Query returned {len(results)} rows")
+ return results
+
+ except Exception as e:
+ print(f"❌ Error executing YDB query: {e}")
+ print(f"❌ Error type: {type(e).__name__}")
+ import traceback
+ print(f"❌ Traceback: {traceback.format_exc()}")
+ return None
+
+
+def get_all_team_data(database_endpoint=None, database_path=None, credentials_path=None, use_yesterday=False):
+ """
+ Get all team data (stats + trends) from YDB in one optimized query.
+
+ Args:
+ database_endpoint (str): YDB database endpoint
+ database_path (str): YDB database path
+ credentials_path (str): Path to service account credentials JSON file
+ use_yesterday (bool): If True, use yesterday's data for development convenience
+
+ Returns:
+ dict: Dictionary with team names as keys and their data, or None if error
+ """
+ # Set up connection
+ print(f"🔍 Setting up YDB connection:")
+ print(f" database_endpoint: {database_endpoint}")
+ print(f" database_path: {database_path}")
+ print(f" credentials_path: {credentials_path}")
+
+ endpoint, path = _setup_ydb_connection(database_endpoint, database_path, credentials_path)
+ if not endpoint or not path:
+ print("❌ Failed to set up YDB connection")
+ return None
+
+ print(f"✅ YDB connection configured: {endpoint} / {path}")
+
+ # Calculate target date
+ if use_yesterday:
+ target_date = datetime.now() - timedelta(days=1)
+ else:
+ target_date = datetime.now()
+
+ yesterday_date = target_date - timedelta(days=1)
+ start_date = target_date - timedelta(days=30)
+
+ print(f"🔍 Date calculation:")
+ print(f" Current datetime.now(): {datetime.now()}")
+ print(f" use_yesterday: {use_yesterday}")
+ print(f" target_date: {target_date}")
+ print(f" start_date: {start_date}")
+ print(f" yesterday_date: {yesterday_date}")
+
+ # Single optimized query for all data
+ all_data_query = f"""
+ SELECT
+ owner,
+ date_window,
+ COUNT(*) as daily_count,
+ SUM(CASE WHEN mute_state_change_date = Date('{target_date.strftime('%Y-%m-%d')}') THEN 1 ELSE 0 END) as today_count
+ FROM `test_results/analytics/test_muted_monitor_mart`
+ WHERE date_window >= Date('{start_date.strftime('%Y-%m-%d')}')
+ AND date_window <= Date('{target_date.strftime('%Y-%m-%d')}')
+ AND is_muted = 1
+ AND branch = 'main'
+ AND build_type = 'relwithdebinfo'
+ GROUP BY owner, date_window
+ ORDER BY owner, date_window
+ """
+
+ # Execute query
+ print(f"🔍 Query details:")
+ print(f" Start date: {start_date.strftime('%Y-%m-%d')}")
+ print(f" Target date: {target_date.strftime('%Y-%m-%d')}")
+ print(f" Endpoint: {endpoint}")
+ print(f" Path: {path}")
+
+ results = _execute_ydb_query(endpoint, path, all_data_query, f"Getting all team data from {start_date.strftime('%Y-%m-%d')} to {target_date.strftime('%Y-%m-%d')}")
+ if results is None:
+ return None
+
+ # Process results
+ team_data = {}
+ base_date = datetime(1970, 1, 1)
+
+ for row in results:
+ owner = row.owner
+ if not owner:
+ continue
+
+ # Handle both "TEAM:@ydb-platform/teamname" and "Unknown" formats
+ if owner.startswith('TEAM:@ydb-platform/'):
+ team_name = owner.split('/')[-1]
+ elif owner == 'Unknown':
+ team_name = 'Unknown'
+ else:
+ # Skip other formats
+ continue
+
+ if team_name not in team_data:
+ team_data[team_name] = {
+ 'stats': {'total': 0, 'today': 0, 'minus_today': 0},
+ 'trend': {}
+ }
+
+ # Convert days since epoch to date
+ date_obj = base_date + timedelta(days=row.date_window)
+ date_str = date_obj.strftime('%Y-%m-%d')
+
+ # Add to trend data
+ team_data[team_name]['trend'][date_str] = row.daily_count
+
+ # Update stats for target date
+ if date_str == target_date.strftime('%Y-%m-%d'):
+ team_data[team_name]['stats']['total'] = row.daily_count
+ team_data[team_name]['stats']['today'] = row.today_count
+
+ # Calculate "minus today" for each team and fix total if needed
+ for team_name, data in team_data.items():
+ trend = data['trend']
+ yesterday_str = yesterday_date.strftime('%Y-%m-%d')
+
+ # If total is 0 but we have trend data, use the latest available value
+ if data['stats']['total'] == 0 and trend:
+ latest_date = max(trend.keys())
+ data['stats']['total'] = trend[latest_date]
+ print(f"🔍 Fixed total for team {team_name}: using {latest_date} value {trend[latest_date]}")
+
+ if yesterday_str in trend:
+ yesterday_total = trend[yesterday_str]
+ today_total = data['stats']['total']
+ today_new = data['stats']['today']
+
+ # minus_today = yesterday_total - (today_total - today_new)
+ data['stats']['minus_today'] = max(0, yesterday_total - (today_total - today_new))
+
+ print(f"📊 Processed data for {len(team_data)} teams")
+ return team_data
+
+
+def get_muted_tests_stats(database_endpoint=None, database_path=None, credentials_path=None, use_yesterday=False):
+ """
+ Get statistics about muted tests from YDB by team.
+
+ Args:
+ database_endpoint (str): YDB database endpoint
+ database_path (str): YDB database path
+ credentials_path (str): Path to service account credentials JSON file
+ use_yesterday (bool): If True, use yesterday's data for development convenience
+
+ Returns:
+ dict: Dictionary with team names as keys and {'total': count, 'today': count} as values, or None if error
+ """
+
+ # Use the optimized function to get all data
+ all_data = get_all_team_data(database_endpoint, database_path, credentials_path, use_yesterday)
+ if all_data is None:
+ return None
+
+ # Extract just the stats part
+ team_stats = {}
+ for team_name, data in all_data.items():
+ team_stats[team_name] = data['stats']
+
+ print(f"📊 Found statistics for {len(team_stats)} teams")
+ return team_stats
+
+
+def get_monthly_trend_data(database_endpoint=None, database_path=None, credentials_path=None, team_name=None, use_yesterday=False):
+ """
+ Get monthly trend data for a specific team.
+
+ Args:
+ database_endpoint (str): YDB database endpoint
+ database_path (str): YDB database path
+ credentials_path (str): Path to service account credentials JSON file
+ team_name (str): Team name to get data for
+ use_yesterday (bool): If True, use yesterday as end date for development convenience
+
+ Returns:
+ dict: Dictionary with dates as keys and counts as values, or None if error
+ """
+ # Use the optimized function to get all data
+ all_data = get_all_team_data(database_endpoint, database_path, credentials_path, use_yesterday)
+ if all_data is None:
+ return None
+
+ # Extract trend data for the specific team
+ if team_name in all_data:
+ trend_data = all_data[team_name]['trend']
+ print(f"📊 Found trend data for {len(trend_data)} days for team '{team_name}'")
+ return trend_data
+ else:
+ print(f"âš ī¸ No data found for team '{team_name}'")
+ return None
+
+
+def create_trend_plot(team_name, trend_data, debug_dir=None):
+ """
+ Create a trend plot for muted tests.
+
+ Args:
+ team_name (str): Team name
+ trend_data (dict): Dictionary with dates as keys and counts as values
+ debug_dir (str): Directory to save debug plot files (if None, debug mode is disabled)
+
+ Returns:
+ str: Base64 encoded image data, or None if error
+ """
+ if not MATPLOTLIB_AVAILABLE:
+ print("❌ Matplotlib not available for plotting")
+ return None
+
+ if not trend_data:
+ print("âš ī¸ No trend data available for plotting")
+ return None
+
+ try:
+ # Prepare data
+ dates = []
+ counts = []
+ for date_str in sorted(trend_data.keys()):
+ dates.append(datetime.strptime(date_str, '%Y-%m-%d'))
+ counts.append(trend_data[date_str])
+
+ # Create plot
+ plt.figure(figsize=(10, 6))
+ plt.plot(dates, counts, marker='o', linewidth=2, markersize=4)
+ plt.title(f'Muted Tests Trend - {team_name}', fontsize=14, fontweight='bold')
+ plt.xlabel('Date', fontsize=12)
+ plt.ylabel('Number of Muted Tests', fontsize=12)
+ plt.grid(True, alpha=0.3)
+ plt.ylim(bottom=0) # Start y-axis from 0
+
+ # Format x-axis
+ plt.gca().xaxis.set_major_formatter(mdates.DateFormatter('%m-%d'))
+ plt.gca().xaxis.set_major_locator(mdates.DayLocator(interval=3))
+ plt.xticks(rotation=45)
+
+ # Add trend line
+ if len(dates) > 1:
+ x_numeric = np.arange(len(dates))
+ z = np.polyfit(x_numeric, counts, 1)
+ p = np.poly1d(z)
+ plt.plot(dates, p(x_numeric), "r--", alpha=0.8, linewidth=2, label='Trend')
+ plt.legend()
+
+ # Add annotation for the last point
+ if dates and counts:
+ last_date = dates[-1]
+ last_count = counts[-1]
+ plt.annotate(f'{last_count}',
+ xy=(last_date, last_count),
+ xytext=(10, 10),
+ textcoords='offset points',
+ bbox=dict(boxstyle='round,pad=0.3', facecolor='yellow', alpha=0.7),
+ arrowprops=dict(arrowstyle='->', connectionstyle='arc3,rad=0'),
+ fontsize=10, fontweight='bold')
+
+ plt.tight_layout()
+
+ # Save to temporary file
+ with tempfile.NamedTemporaryFile(suffix='.png', delete=False) as tmp_file:
+ plt.savefig(tmp_file.name, dpi=150, bbox_inches='tight')
+ tmp_path = tmp_file.name
+
+ print(f"📈 Created trend plot for {team_name}: {tmp_path}")
+ print(f"📁 File size: {os.path.getsize(tmp_path)} bytes")
+
+ # Save to debug directory if requested
+ if debug_dir:
+ os.makedirs(debug_dir, exist_ok=True)
+ debug_path = os.path.join(debug_dir, f"trend_{team_name}_{datetime.now().strftime('%Y%m%d_%H%M%S')}.png")
+
+ # Copy to debug directory
+ shutil.copy2(tmp_path, debug_path)
+ print(f"🔍 Debug plot saved to: {debug_path}")
+
+ # Read and encode as base64
+ with open(tmp_path, 'rb') as f:
+ image_data = f.read()
+
+ print(f"📊 Base64 data length: {len(base64.b64encode(image_data).decode('utf-8'))} characters")
+
+ # Clean up temporary file (but keep debug file)
+ os.unlink(tmp_path)
+ plt.close()
+
+ # Encode as base64
+ base64_data = base64.b64encode(image_data).decode('utf-8')
+ return base64_data
+
+ except Exception as e:
+ print(f"❌ Error creating trend plot: {e}")
+ return None
+
def parse_team_issues(content):
"""
@@ -65,7 +475,7 @@ def parse_team_issues(content):
def escape_markdown(text):
"""
- Escape special Markdown characters for Telegram, but preserve link structure.
+ Escape special MarkdownV2 characters for Telegram, but preserve bold formatting and link structure.
Args:
text (str): Text to escape
@@ -73,14 +483,71 @@ def escape_markdown(text):
Returns:
str: Escaped text
"""
- special_chars = ['*', '~', '>', '#', '=', '|', '{', '}', '!']
+ # For MarkdownV2, we need to escape these characters: _ * [ ] ( ) ~ ` > # + - = | { } . !
+ # But we want to preserve * for bold formatting, [ ] ( ) for links, and content inside backticks
+
+ # First, protect inline code by temporarily replacing them
+ code_pattern = r'`([^`]+)`'
+ code_blocks = []
+
+ def replace_code(match):
+ code_blocks.append(match.group(1))
+ return f"CODEPLACEHOLDER{len(code_blocks)-1}N"
+
+ # Replace all inline code with placeholders
+ text = re.sub(code_pattern, replace_code, text)
+
+ # Then protect links by temporarily replacing them
+ link_pattern = r'\[([^\]]+)\]\(([^)]+)\)'
+ links = []
+
+ def replace_link(match):
+ links.append((match.group(1), match.group(2)))
+ return f"LINKPLACEHOLDER{len(links)-1}N"
+
+ # Replace all links with placeholders
+ text = re.sub(link_pattern, replace_link, text)
+
+ # Also protect standalone URLs (http/https) by temporarily replacing them
+ url_pattern = r'https?://[^\s\)]+'
+ urls = []
+
+ def replace_url(match):
+ urls.append(match.group(0))
+ return f"URLPLACEHOLDER{len(urls)-1}N"
+
+ # Replace all standalone URLs with placeholders
+ text = re.sub(url_pattern, replace_url, text)
+
+ # Escape special characters for MarkdownV2 (single backslash)
+ # Characters that need escaping: _ * [ ] ( ) ~ ` > # + - = | { } . !
+ special_chars = ['_', '~', '`', '>', '#', '+', '-', '=', '|', '{', '}', '.', '!', '(', ')']
for char in special_chars:
text = text.replace(char, f'\\{char}')
+
+ # Restore standalone URLs (they should NOT be escaped)
+ for i, url in enumerate(urls):
+ text = text.replace(f"URLPLACEHOLDER{i}N", url)
+
+ # Restore links, escaping special characters in link text but NOT in URLs
+ for i, (link_text, link_url) in enumerate(links):
+ # Escape special characters in link text only
+ escaped_text = link_text
+ for char in special_chars:
+ escaped_text = escaped_text.replace(char, f'\\{char}')
+
+ # URLs should NOT be escaped (Telegram handles them correctly)
+ text = text.replace(f"LINKPLACEHOLDER{i}N", f"[{escaped_text}]({link_url})")
+
+ # Restore inline code (content should NOT be escaped)
+ for i, code_content in enumerate(code_blocks):
+ text = text.replace(f"CODEPLACEHOLDER{i}N", f"`{code_content}`")
+
return text
-def format_team_message(team_name, issues, team_responsible=None):
+def format_team_message(team_name, issues, team_responsible=None, muted_stats=None, show_diff=False):
"""
Format message for a specific team.
@@ -88,6 +555,8 @@ def format_team_message(team_name, issues, team_responsible=None):
team_name (str): Team name
issues (list): List of issues for the team
team_responsible (dict): Dictionary mapping team names to responsible usernames
+ muted_stats (dict): Dictionary with team names as keys and {'total': count, 'today': count} as values
+ show_diff (bool): Whether to show +/- statistics
Returns:
str: Formatted message
@@ -100,9 +569,48 @@ def format_team_message(team_name, issues, team_responsible=None):
# Start with title and team tag (replace - with _ in tag)
team_tag = team_name.replace('-', '')
- message = f"🔇 **{current_date} new muted tests for [{team_name}](https://github.com/orgs/ydb-platform/teams/{team_name})** #{team_tag}\n\n"
+ # Create team URL and escape it
+ team_url = f"{GITHUB_ORG_TEAMS_URL}/{team_name}"
+ message = f"🔇 *{current_date} new muted tests in `main` for [{team_name}]({team_url})* #{team_tag}\n\n"
+
+ for issue in issues:
+ # Extract issue number from URL for compact display
+ issue_number = issue['url'].split('/')[-1] if '/' in issue['url'] else issue['url']
+
+ # Remove "in main" from title if present and "Mute " prefix
+ title = issue['title']
+ if title.endswith(' in main'):
+ title = title[:-8] # Remove " in main"
+ if title.startswith('Mute '):
+ title = title[5:] # Remove "Mute " prefix
+
+ # Wrap title in backticks (will be escaped later with the whole message)
+ message += f" đŸŽ¯ `{title}` [#{issue_number}]({issue['url']})\n"
- # Add responsible users on new line with "fyi:" prefix
+ # Add muted tests statistics for this specific team if available (moved to end)
+ if muted_stats and team_name in muted_stats:
+ team_stats = muted_stats[team_name]
+ total = team_stats['total']
+ today = team_stats['today']
+ minus_today = team_stats.get('minus_today', 0)
+
+ # Create dashboard URL for the team
+ dashboard_url = DATALENS_DASHBOARD_URL.format(team_name=team_name)
+
+ # Format statistics with color coding and emojis
+ if show_diff:
+ if today > 0 and minus_today > 0:
+ message += f"\n📊 *[Total muted tests: {total}]({dashboard_url}) 🔴+{today} muted /đŸŸĸ-{minus_today} unmuted*"
+ elif today > 0:
+ message += f"\n📊 *[Total muted tests: {total}]({dashboard_url}) 🔴+{today} muted*"
+ elif minus_today > 0:
+ message += f"\n📊 *[Total muted tests: {total}]({dashboard_url}) đŸŸĸ-{minus_today} unmuted*"
+ else:
+ message += f"\n📊 *[Total muted tests: {total}]({dashboard_url})*"
+ else:
+ message += f"\n📊 *[Total muted tests: {total}]({dashboard_url})*"
+
+ # Add responsible users on new line with "fyi:" prefix (moved after statistics)
if team_responsible and team_name in team_responsible:
responsible = team_responsible[team_name]
# Handle both single responsible and list of responsibles
@@ -110,12 +618,7 @@ def format_team_message(team_name, issues, team_responsible=None):
responsible_str = " ".join(f"@{r.replace('_', '\\_')}" if not r.startswith('@') else r.replace('_', '\\_') for r in responsible)
else:
responsible_str = f"@{responsible.replace('_', '\\_')}" if not responsible.startswith('@') else responsible.replace('_', '\\_')
- message += f"fyi: {responsible_str}\n\n"
-
- for issue in issues:
- # Escape the title for Markdown and wrap in backticks
- escaped_title = escape_markdown(issue['title'])
- message += f" - đŸŽ¯ [{issue['url']}]({issue['url']}) - `{escaped_title}`\n"
+ message += f"\n\nfyi: {responsible_str}"
# Add empty line at the end for better readability
message += "\n"
@@ -205,7 +708,7 @@ def get_team_config(team_name, team_channels):
return None, None, None
-def send_team_messages(teams, bot_token, delay=2, max_retries=5, retry_delay=10, team_channels=None, dry_run=False):
+def send_team_messages(teams, bot_token, delay=2, max_retries=5, retry_delay=10, team_channels=None, dry_run=False, muted_stats=None, include_plots=False, ydb_config=None, debug_plots_dir=None, all_team_data=None, show_diff=False):
"""
Send separate messages for each team.
@@ -217,6 +720,12 @@ def send_team_messages(teams, bot_token, delay=2, max_retries=5, retry_delay=10,
retry_delay (int): Delay between retry attempts in seconds
team_channels (dict): Dictionary mapping team names to their specific channel configs
dry_run (bool): If True, only print messages without sending to Telegram
+ muted_stats (dict): Dictionary with team names as keys and {'total': count, 'today': count} as values
+ include_plots (bool): If True, include trend plots in messages
+ ydb_config (dict): YDB configuration for trend data
+ debug_plots_dir (str): Directory to save debug plot files (if None, debug mode is disabled)
+ all_team_data (dict): Pre-fetched team data to avoid repeated queries
+ show_diff (bool): Whether to show +/- statistics in messages
"""
total_teams = len(teams)
@@ -241,11 +750,20 @@ def send_team_messages(teams, bot_token, delay=2, max_retries=5, retry_delay=10,
continue
# Format message
- message = format_team_message(team_name, issues, team_responsible)
+ message = format_team_message(team_name, issues, team_responsible, muted_stats, show_diff)
if not message.strip():
continue
+ # Escape the entire message for MarkdownV2
+ message = escape_markdown(message)
+
+ # Print final message before sending
+ print(f"🔍 Final message for {team_name}:")
+ print("-" * 80)
+ print(message)
+ print("-" * 80)
+
if dry_run:
print(f"\n--- Team: {team_name} ---")
print(f"📨 Channel: {team_chat_id}" + (f" (thread {team_thread_id})" if team_thread_id else ""))
@@ -254,11 +772,86 @@ def send_team_messages(teams, bot_token, delay=2, max_retries=5, retry_delay=10,
else:
print(f"📨 Sending message for team: {team_name} ({len(issues)} issues)")
- if send_telegram_message(bot_token, team_chat_id, message, "Markdown", team_thread_id, True, max_retries, retry_delay):
- sent_count += 1
- print(f"✅ Message sent for team: {team_name}")
+ # Get trend plot if requested
+ plot_data = None
+ print(f"🔍 Plot settings: include_plots={include_plots}, ydb_config={ydb_config is not None}, MATPLOTLIB_AVAILABLE={MATPLOTLIB_AVAILABLE}")
+
+ if include_plots and MATPLOTLIB_AVAILABLE:
+ # Use pre-fetched data if available, otherwise fetch on demand
+ if all_team_data and team_name in all_team_data:
+ trend_data = all_team_data[team_name]['trend']
+ print(f"📊 Using cached trend data for {team_name}: {len(trend_data)} days")
+ elif ydb_config:
+ print(f"📊 Getting trend data for team: {team_name}")
+ trend_data = get_monthly_trend_data(
+ database_endpoint=ydb_config.get('endpoint'),
+ database_path=ydb_config.get('path'),
+ credentials_path=ydb_config.get('credentials'),
+ team_name=team_name,
+ use_yesterday=ydb_config.get('use_yesterday', False)
+ )
+ else:
+ trend_data = None
+
+ if trend_data:
+ plot_data = create_trend_plot(team_name, trend_data, debug_plots_dir)
+ if plot_data:
+ print(f"📈 Created trend plot for team: {team_name} (data length: {len(plot_data)})")
+ else:
+ print(f"âš ī¸ Could not create trend plot for team: {team_name}")
+ else:
+ print(f"âš ī¸ No trend data available for team: {team_name}")
+ elif include_plots and not MATPLOTLIB_AVAILABLE:
+ print(f"âš ī¸ Matplotlib not available, skipping plot for team: {team_name}")
+ elif include_plots and not ydb_config:
+ print(f"âš ī¸ YDB config not available, skipping plot for team: {team_name}")
+ else:
+ print(f"â„šī¸ Plots disabled for team: {team_name}")
+
+ # Send message with or without plot
+ if plot_data:
+ # Save plot to temporary file and send as photo
+ with tempfile.NamedTemporaryFile(suffix='.png', delete=False) as tmp_file:
+ tmp_file.write(base64.b64decode(plot_data))
+ tmp_path = tmp_file.name
+
+ print(f"📁 Saved plot to temporary file: {tmp_path}")
+ print(f"📁 File size: {os.path.getsize(tmp_path)} bytes")
+
+ # Also save to debug directory for final file if requested
+ if debug_plots_dir:
+ os.makedirs(debug_plots_dir, exist_ok=True)
+ final_debug_path = os.path.join(debug_plots_dir, f"final_{team_name}_{datetime.now().strftime('%Y%m%d_%H%M%S')}.png")
+ shutil.copy2(tmp_path, final_debug_path)
+ print(f"🔍 Final debug plot saved to: {final_debug_path}")
+
+ try:
+ print(f"📤 Sending message with plot for team: {team_name}")
+ print(f"📤 Chat ID: {team_chat_id}, Thread ID: {team_thread_id}")
+ print(f"📤 Photo path: {tmp_path}")
+ print(f"📤 Message length: {len(message)} characters")
+
+ if send_telegram_message(bot_token, team_chat_id, message, "MarkdownV2", team_thread_id, True, max_retries, retry_delay, photo_path=tmp_path):
+ sent_count += 1
+ print(f"✅ Message with plot sent for team: {team_name}")
+ else:
+ print(f"❌ Failed to send message with plot for team: {team_name} after {max_retries} retries")
+ finally:
+ # Clean up temporary file
+ if os.path.exists(tmp_path):
+ os.unlink(tmp_path)
+ print(f"đŸ—‘ī¸ Cleaned up temporary file: {tmp_path}")
else:
- print(f"❌ Failed to send message for team: {team_name} after {max_retries} retries")
+ # Send regular message
+ print(f"📤 Sending regular message for team: {team_name}")
+ print(f"📤 Chat ID: {team_chat_id}, Thread ID: {team_thread_id}")
+ print(f"📤 Message length: {len(message)} characters")
+
+ if send_telegram_message(bot_token, team_chat_id, message, "MarkdownV2", team_thread_id, True, max_retries, retry_delay):
+ sent_count += 1
+ print(f"✅ Message sent for team: {team_name}")
+ else:
+ print(f"❌ Failed to send message for team: {team_name} after {max_retries} retries")
# Add delay between messages
if sent_count < total_teams:
@@ -372,11 +965,248 @@ def load_team_channels(team_channels_json):
return None
+def send_period_updates(period, bot_token, team_channels, ydb_config, delay=2, max_retries=5, retry_delay=10, dry_run=False, debug_plots_dir=None):
+ """
+ Send periodic trend updates for all teams.
+
+ Args:
+ period (str): Period type ('week' or 'month')
+ bot_token (str): Telegram bot token
+ team_channels (dict): Dictionary mapping team names to their channel configurations
+ ydb_config (dict): YDB configuration
+ delay (int): Delay between messages in seconds
+ max_retries (int): Maximum number of retry attempts
+ retry_delay (int): Delay between retry attempts in seconds
+ dry_run (bool): If True, only print messages without sending to Telegram
+ debug_plots_dir (str): Directory to save debug plot files
+
+ Returns:
+ bool: True if successful, False otherwise
+ """
+ print(f"📊 Starting {period}ly trend updates...")
+
+ # Get all team data for trends
+ all_team_data = get_all_team_data(
+ database_endpoint=ydb_config.get('endpoint'),
+ database_path=ydb_config.get('path'),
+ credentials_path=ydb_config.get('credentials'),
+ use_yesterday=ydb_config.get('use_yesterday', False)
+ )
+
+ if not all_team_data:
+ print("❌ Could not fetch team data for trend updates")
+ return False
+
+ # Get teams from data (all teams that have data)
+ teams_from_data = list(all_team_data.keys())
+ if not teams_from_data:
+ print("❌ No teams found in data")
+ return False
+
+ # Filter out blacklisted teams
+ teams_to_process = [team for team in teams_from_data if team not in PERIOD_UPDATE_BLACKLIST]
+ blacklisted_count = len(teams_from_data) - len(teams_to_process)
+
+ if blacklisted_count > 0:
+ print(f"â­ī¸ Skipping {blacklisted_count} blacklisted teams: {', '.join(team for team in teams_from_data if team in PERIOD_UPDATE_BLACKLIST)}")
+
+ print(f"📤 Sending {period}ly updates for {len(teams_to_process)} teams from data...")
+
+ success_count = 0
+ total_teams = len(teams_to_process)
+
+ for team_name in teams_to_process:
+
+ # Get team channel configuration
+ team_responsible, team_chat_id, team_thread_id = get_team_config(team_name, team_channels)
+
+ # If team not found in config, use default channel
+ if not team_chat_id and team_channels:
+ default_channel_name = team_channels.get('default_channel')
+ if default_channel_name and 'channels' in team_channels:
+ if default_channel_name in team_channels['channels']:
+ team_chat_id, team_thread_id = parse_chat_and_thread_id(team_channels['channels'][default_channel_name])
+ print(f"📨 Using default channel '{default_channel_name}' for team {team_name}: {team_chat_id}")
+
+ # Determine channel name for logging
+ if team_channels and 'teams' in team_channels and team_name in team_channels['teams']:
+ team_config = team_channels['teams'][team_name]
+ team_channel_name = team_config.get('channel', team_channels.get('default_channel', 'default'))
+ else:
+ team_channel_name = team_channels.get('default_channel', 'default') if team_channels else 'default'
+
+ if not team_chat_id:
+ print(f"âš ī¸ No channel configuration for team: {team_name} (skipping)")
+ continue
+
+ # Create trend message
+ period_title = "Weekly Muted Tests Report" if period == "week" else "Monthly Muted Tests Report"
+ team_tag = team_name.replace('-', '')
+ team_url = f"{GITHUB_ORG_TEAMS_URL}/{team_name}"
+ message = f"📈 *{period_title}* for team [{team_name}]({team_url}) #{team_tag}\n\n"
+
+ # Add trend statistics if available
+ if team_name in all_team_data:
+ team_stats = all_team_data[team_name]['stats']
+ trend_data = all_team_data[team_name]['trend']
+ total = team_stats['total']
+ dashboard_url = DATALENS_DASHBOARD_URL.format(team_name=team_name)
+
+ # Calculate period change
+ period_days = 7 if period == "week" else 30
+ # Use the same date logic as in get_all_team_data
+ current_date = datetime.now() - timedelta(days=1) if ydb_config.get('use_yesterday', False) else datetime.now()
+ previous_date = current_date - timedelta(days=period_days)
+
+ current_date_str = current_date.strftime('%Y-%m-%d')
+ previous_date_str = previous_date.strftime('%Y-%m-%d')
+
+ current_count = trend_data.get(current_date_str, 0)
+ previous_count = trend_data.get(previous_date_str, 0)
+
+ # Check if we have data for both dates to calculate meaningful change
+ has_current_data = current_date_str in trend_data
+ has_previous_data = previous_date_str in trend_data
+
+ # Debug information
+ print(f"🔍 Debug for team {team_name}:")
+ print(f" Current date: {current_date_str}, count: {current_count}, has_data: {has_current_data}")
+ print(f" Previous date: {previous_date_str}, count: {previous_count}, has_data: {has_previous_data}")
+ print(f" Available dates in trend_data: {sorted(trend_data.keys())[-5:]}") # Last 5 dates
+
+ message += f"📊 *[Total muted tests: {total}]({dashboard_url})*\n\n"
+
+ # Add change information only if we have data for both dates
+ if has_current_data and has_previous_data:
+ change = current_count - previous_count
+ print(f" Change: {change} (current - previous)")
+ if change > 0:
+ message += f"🔴 +{change} muted tests in last {period_days} days\n\n"
+ elif change < 0:
+ message += f"đŸŸĸ {change} muted tests in last {period_days} days\n\n"
+ else:
+ message += f"âšĒ No change in last {period_days} days\n\n"
+ else:
+ print(f" Cannot calculate change: missing data for current={has_current_data}, previous={has_previous_data}")
+
+ # Try to find the earliest available data for comparison
+ if trend_data and has_current_data:
+ available_dates = sorted(trend_data.keys())
+ if len(available_dates) > 1:
+ earliest_date = available_dates[0]
+ earliest_count = trend_data[earliest_date]
+ days_span = (datetime.strptime(current_date_str, '%Y-%m-%d') - datetime.strptime(earliest_date, '%Y-%m-%d')).days
+ change = current_count - earliest_count
+ print(f" Using earliest available data: {earliest_date} ({earliest_count}) vs {current_date_str} ({current_count}) = {change} over {days_span} days")
+
+ if change > 0:
+ message += f"🔴 +{change} muted tests since {earliest_date} ({days_span} days ago)\n\n"
+ elif change < 0:
+ message += f"đŸŸĸ {change} muted tests since {earliest_date} ({days_span} days ago)\n\n"
+ else:
+ message += f"âšĒ No change since {earliest_date} ({days_span} days ago)\n\n"
+ else:
+ message += f"â„šī¸ No historical data available for comparison\n\n"
+ else:
+ if not has_previous_data:
+ message += f"â„šī¸ No data available for comparison ({period_days} days ago)\n\n"
+ else:
+ message += f"â„šī¸ Insufficient data for trend analysis\n\n"
+
+ # Add responsible users if available
+ if team_responsible and team_name in team_responsible:
+ responsible = team_responsible[team_name]
+ # Handle both single responsible and list of responsibles
+ if isinstance(responsible, list):
+ responsible_str = " ".join(f"@{r.replace('_', '\\_')}" if not r.startswith('@') else r.replace('_', '\\_') for r in responsible)
+ else:
+ responsible_str = f"@{responsible.replace('_', '\\_')}" if not responsible.startswith('@') else responsible.replace('_', '\\_')
+ message += f"fyi: {responsible_str}\n\n"
+
+ # Escape the entire message for MarkdownV2
+ message = escape_markdown(message)
+
+ # Print final message before sending
+ print(f"🔍 Final {period}ly message for {team_name}:")
+ print("-" * 80)
+ print(message)
+ print("-" * 80)
+
+ if dry_run:
+ print(f"📋 [DRY RUN] Team: {team_name}")
+ print(f"📋 Channel: {team_channel_name} ({team_chat_id})")
+ print(f"📋 Thread: {team_thread_id}")
+ print("📋 Message:")
+ print("-" * 80)
+ print(message)
+ print("-" * 80)
+ print()
+ success_count += 1
+ else:
+ print(f"📨 Sending {period}ly update for team: {team_name}")
+
+ # Get trend data for this team
+ trend_data = all_team_data.get(team_name, {}).get('trend', {})
+
+ if trend_data and MATPLOTLIB_AVAILABLE:
+ # Create trend plot
+ plot_data = create_trend_plot(team_name, trend_data, debug_plots_dir)
+
+ if plot_data:
+ # Send message with plot
+ with tempfile.NamedTemporaryFile(suffix='.png', delete=False) as tmp_file:
+ tmp_file.write(base64.b64decode(plot_data))
+ tmp_path = tmp_file.name
+
+ try:
+ print(f"📤 Sending trend plot for team: {team_name}")
+ if send_telegram_message(bot_token, team_chat_id, message, "MarkdownV2", team_thread_id, True, max_retries, retry_delay, photo_path=tmp_path):
+ success_count += 1
+ print(f"✅ Trend update sent for team: {team_name}")
+ else:
+ print(f"❌ Failed to send trend update for team: {team_name}")
+ finally:
+ # Clean up temporary file
+ if os.path.exists(tmp_path):
+ os.unlink(tmp_path)
+ else:
+ # Fallback to text message
+ print(f"âš ī¸ Could not create plot, sending text message for team: {team_name}")
+ if send_telegram_message(bot_token, team_chat_id, message, "MarkdownV2", team_thread_id, True, max_retries, retry_delay):
+ success_count += 1
+ print(f"✅ Text update sent for team: {team_name}")
+ else:
+ print(f"❌ Failed to send text update for team: {team_name}")
+ else:
+ # Send text message only
+ print(f"📤 Sending text update for team: {team_name}")
+ if send_telegram_message(bot_token, team_chat_id, message, "MarkdownV2", team_thread_id, True, max_retries, retry_delay):
+ success_count += 1
+ print(f"✅ Text update sent for team: {team_name}")
+ else:
+ print(f"❌ Failed to send text update for team: {team_name}")
+
+ # Add delay between messages
+ if success_count < total_teams:
+ time.sleep(delay)
+
+ if dry_run:
+ print(f"🎉 Dry run completed: {success_count}/{total_teams} {period}ly updates formatted!")
+ if blacklisted_count > 0:
+ print(f"📊 Summary: {len(teams_from_data)} total teams, {blacklisted_count} blacklisted, {total_teams} processed")
+ else:
+ print(f"🎉 Sent {success_count}/{total_teams} {period}ly updates successfully!")
+ if blacklisted_count > 0:
+ print(f"📊 Summary: {len(teams_from_data)} total teams, {blacklisted_count} blacklisted, {total_teams} processed, {success_count} sent")
+
+ return success_count == total_teams
+
+
def main():
parser = argparse.ArgumentParser(description="Parse team issues and send separate messages for each team")
# Required arguments
- parser.add_argument('--file', required=True, help='Path to file with formatted results')
+ parser.add_argument('--file', help='Path to file with formatted results (required for --on-mute-change-update mode)')
parser.add_argument('--bot-token', help='Telegram bot token (or use TELEGRAM_BOT_TOKEN env var)')
parser.add_argument('--team-channels', required=True, help='JSON string mapping teams to their channel configurations (or use TEAM_CHANNELS env var)')
@@ -388,8 +1218,27 @@ def main():
parser.add_argument('--max-retries', type=int, default=5, help='Maximum number of retry attempts for failed messages (default: 5)')
parser.add_argument('--retry-delay', type=int, default=10, help='Delay between retry attempts in seconds (default: 10)')
+ # Mode selection (mutually exclusive)
+ mode_group = parser.add_mutually_exclusive_group(required=True)
+ mode_group.add_argument('--on-mute-change-update', action='store_true', help='Default mode: send updates about new muted tests')
+ mode_group.add_argument('--period-update', choices=['week', 'month'], help='Send periodic trend updates (week or month)')
+
+ # YDB arguments for muted tests statistics
+ parser.add_argument('--ydb-endpoint', help='YDB database endpoint (or use YDB_ENDPOINT env var)')
+ parser.add_argument('--ydb-database', help='YDB database path (or use YDB_DATABASE env var)')
+ parser.add_argument('--ydb-credentials', help='Path to YDB service account credentials JSON file (or use YDB_SERVICE_ACCOUNT_KEY_FILE_CREDENTIALS env var)')
+ parser.add_argument('--no-stats', action='store_true', help='Skip fetching muted tests statistics from YDB')
+ parser.add_argument('--use-yesterday', action='store_true', help='Use yesterday\'s data for development convenience')
+ parser.add_argument('--include-plots', action='store_true', help='Include trend plots in messages (requires matplotlib)')
+ parser.add_argument('--debug-plots-dir', help='Directory to save debug plot files (enables debug mode)')
+
args = parser.parse_args()
+ # Validate mode-specific requirements
+ if args.on_mute_change_update and not args.file:
+ print("❌ --file is required for --on-mute-change-update mode")
+ sys.exit(1)
+
# Get bot token
bot_token = args.bot_token or os.getenv('TELEGRAM_BOT_TOKEN')
@@ -405,6 +1254,60 @@ def main():
print(f"📋 Loaded channel configurations for {len(team_channels.get('teams', {}))} teams")
+ # Handle period update mode
+ if args.period_update:
+ # Prepare YDB config for period updates
+ ydb_config = {
+ 'endpoint': args.ydb_endpoint,
+ 'path': args.ydb_database,
+ 'credentials': args.ydb_credentials,
+ 'use_yesterday': args.use_yesterday
+ }
+
+ # Check if we need Telegram connection (not for dry run)
+ if not args.dry_run:
+ if not bot_token:
+ print("❌ Bot token not provided. Use --bot-token or set TELEGRAM_BOT_TOKEN environment variable")
+ sys.exit(1)
+
+ # Send period updates
+ success = send_period_updates(
+ period=args.period_update,
+ bot_token=bot_token,
+ team_channels=team_channels,
+ ydb_config=ydb_config,
+ delay=args.delay,
+ max_retries=args.max_retries,
+ retry_delay=args.retry_delay,
+ dry_run=args.dry_run,
+ debug_plots_dir=args.debug_plots_dir
+ )
+
+ if success:
+ print("✅ Period updates completed successfully")
+ sys.exit(0)
+ else:
+ print("❌ Period updates failed")
+ sys.exit(1)
+
+ # Handle on-mute-change-update mode (default mode)
+ # Get muted tests statistics if not disabled
+ muted_stats = None
+ if not args.no_stats:
+ print("📊 Fetching muted tests statistics from YDB...")
+ muted_stats = get_muted_tests_stats(
+ database_endpoint=args.ydb_endpoint,
+ database_path=args.ydb_database,
+ credentials_path=args.ydb_credentials,
+ use_yesterday=args.use_yesterday
+ )
+ if muted_stats:
+ print(f"✅ Statistics loaded for {len(muted_stats)} teams")
+ else:
+ print("âš ī¸ Could not load statistics, continuing without stats")
+ else:
+ print("â­ī¸ Skipping statistics fetch (--no-stats flag)")
+
# Check if we need Telegram connection (not for dry run)
if not args.dry_run or args.test_connection:
if not bot_token:
@@ -479,8 +1382,48 @@ def main():
print(f" - {team_name}: {len(issues)} issues{responsible_info}{channel_info}")
+ # Prepare YDB config and get all team data if needed
+ ydb_config = None
+ all_team_data = None
+
+ if args.include_plots and not args.no_stats:
+ ydb_config = {
+ 'endpoint': args.ydb_endpoint,
+ 'path': args.ydb_database,
+ 'credentials': args.ydb_credentials,
+ 'use_yesterday': args.use_yesterday
+ }
+
+ # Get all team data in one optimized query
+ print("📊 Fetching all team data in one optimized query...")
+ all_team_data = get_all_team_data(
+ database_endpoint=args.ydb_endpoint,
+ database_path=args.ydb_database,
+ credentials_path=args.ydb_credentials,
+ use_yesterday=args.use_yesterday
+ )
+
+ if all_team_data:
+ print(f"✅ Successfully fetched data for {len(all_team_data)} teams")
+ else:
+ print("âš ī¸ Could not fetch team data, will use individual queries if needed")
+
# Send messages (or show in dry run)
- send_team_messages(teams, bot_token, args.delay, args.max_retries, args.retry_delay, team_channels, args.dry_run)
+ send_team_messages(
+ teams,
+ bot_token,
+ args.delay,
+ args.max_retries,
+ args.retry_delay,
+ team_channels,
+ args.dry_run,
+ muted_stats,
+ args.include_plots,
+ ydb_config,
+ args.debug_plots_dir,
+ all_team_data,
+ MUTE_UPDATE_SHOW_DIFF
+ )
if __name__ == "__main__":
diff --git a/.github/scripts/telegram/send_telegram_message.py b/.github/scripts/telegram/send_telegram_message.py
index d16fd348ca0..5ac64808408 100644
--- a/.github/scripts/telegram/send_telegram_message.py
+++ b/.github/scripts/telegram/send_telegram_message.py
@@ -12,50 +12,50 @@ import time
from pathlib import Path
-def send_telegram_message(bot_token, chat_id, message_or_file, parse_mode="Markdown", message_thread_id=None, disable_web_page_preview=True, max_retries=5, retry_delay=10, delay=1):
+def _read_content(message_or_file):
"""
- Send a message to Telegram channel with retry mechanism.
- Can accept either text message or file path.
+ Read content from message string or file.
+
+ Args:
+ message_or_file (str): Message text or path to file to send
+
+ Returns:
+ str: Content to send, or None if error
+ """
+ # Try to open as file first, if it fails, treat as message
+ try:
+ with open(message_or_file, 'r', encoding='utf-8') as f:
+ return f.read()
+ except (OSError, IOError, ValueError, UnicodeDecodeError):
+ # It's a text message, not a file (or file can't be read as text)
+ return message_or_file
+
+
+def _send_chunked_messages(bot_token, chat_id, content, parse_mode, message_thread_id, disable_web_page_preview, max_retries, retry_delay, delay, max_length=4000):
+ """
+ Send chunked text messages.
Args:
bot_token (str): Telegram bot token
chat_id (str): Telegram chat ID
- message_or_file (str): Message text or path to file to send
+ content (str): Content to send
parse_mode (str): Parse mode for message formatting
message_thread_id (int, optional): Thread ID for group messages
disable_web_page_preview (bool): Disable web page preview for links
max_retries (int): Maximum number of retry attempts
retry_delay (int): Delay between retry attempts in seconds
delay (int): Delay between message chunks in seconds
+ max_length (int): Maximum length per chunk
Returns:
- bool: True if successful, False otherwise
+ bool: True if all chunks sent successfully, False otherwise
"""
- # Check if message_or_file is a file path
- # If the string is too long, it's definitely not a file path
- if len(message_or_file) > 255:
- # It's definitely a text message, not a file path
- content = message_or_file
- else:
- file_path = Path(message_or_file)
- if file_path.exists() and file_path.is_file():
- # It's a file, read its content
- try:
- with open(file_path, 'r', encoding='utf-8') as f:
- content = f.read()
- except Exception as e:
- print(f"❌ Error reading file {file_path}: {e}")
- return False
- else:
- # It's a text message
- content = message_or_file
-
if not content.strip():
print(f"âš ī¸ Content is empty")
return True
# Split message into chunks if needed
- chunks = split_message(content)
+ chunks = split_message(content, max_length)
print(f"📤 Sending {len(chunks)} message(s)...")
success_count = 0
@@ -77,6 +77,104 @@ def send_telegram_message(bot_token, chat_id, message_or_file, parse_mode="Markd
return False
+def _send_photo_with_chunked_caption(bot_token, chat_id, photo_path, content, parse_mode, message_thread_id, disable_web_page_preview, max_retries, retry_delay, delay):
+ """
+ Send photo with chunked caption.
+
+ Args:
+ bot_token (str): Telegram bot token
+ chat_id (str): Telegram chat ID
+ photo_path (str): Path to photo file
+ content (str): Caption content
+ parse_mode (str): Parse mode for message formatting
+ message_thread_id (int, optional): Thread ID for group messages
+ disable_web_page_preview (bool): Disable web page preview for links
+ max_retries (int): Maximum number of retry attempts
+ retry_delay (int): Delay between retry attempts in seconds
+ delay (int): Delay between message chunks in seconds
+
+ Returns:
+ bool: True if all chunks sent successfully, False otherwise
+ """
+ photo_file = Path(photo_path)
+ if not photo_file.exists() or not photo_file.is_file():
+ print(f"❌ Photo file not found: {photo_path}")
+ return False
+
+ print(f"📷 Sending photo: {photo_path}")
+
+ # Split message into chunks for photo caption
+ chunks = split_message(content, max_length=1024) # Telegram caption limit is 1024 characters
+ print(f"📤 Sending photo with {len(chunks)} message chunk(s)...")
+
+ success_count = 0
+ for i, chunk in enumerate(chunks, 1):
+ print(f"📨 Sending photo chunk {i}/{len(chunks)}...")
+
+ if i == 1:
+ # First chunk goes with photo
+ if _send_photo(bot_token, chat_id, photo_path, chunk, parse_mode, message_thread_id, max_retries, retry_delay):
+ success_count += 1
+ print(f"✅ Photo with caption sent successfully!")
+ else:
+ print(f"❌ Failed to send photo with caption")
+ else:
+ # Subsequent chunks as regular messages
+ if _send_single_message(bot_token, chat_id, chunk, parse_mode, message_thread_id, disable_web_page_preview, max_retries, retry_delay):
+ success_count += 1
+ print(f"✅ Text chunk {i} sent successfully!")
+ else:
+ print(f"❌ Failed to send text chunk {i}")
+
+ # Add delay between messages (except for the last one)
+ if i < len(chunks):
+ time.sleep(delay)
+
+ if success_count == len(chunks):
+ print(f"✅ All {success_count} message(s) sent successfully!")
+ return True
+ else:
+ print(f"âš ī¸ Only {success_count}/{len(chunks)} message(s) sent successfully")
+ return False
+
+
+def send_telegram_message(bot_token, chat_id, message_or_file, parse_mode="MarkdownV2", message_thread_id=None, disable_web_page_preview=True, max_retries=5, retry_delay=10, delay=1, photo_path=None):
+ """
+ Send a message to Telegram channel with retry mechanism.
+ Can accept either text message or file path.
+
+ Args:
+ bot_token (str): Telegram bot token
+ chat_id (str): Telegram chat ID
+ message_or_file (str): Message text or path to file to send
+ parse_mode (str): Parse mode for message formatting
+ message_thread_id (int, optional): Thread ID for group messages
+ disable_web_page_preview (bool): Disable web page preview for links
+ max_retries (int): Maximum number of retry attempts
+ retry_delay (int): Delay between retry attempts in seconds
+ delay (int): Delay between message chunks in seconds
+ photo_path (str, optional): Path to photo file to send
+
+ Returns:
+ bool: True if successful, False otherwise
+ """
+ print(f"🔍 send_telegram_message called with photo_path: {photo_path}")
+ print(f"🔍 message_or_file length: {len(message_or_file) if message_or_file else 'None'}")
+ print(f"🔍 chat_id: {chat_id}, message_thread_id: {message_thread_id}")
+
+ # Read content from message string or file
+ content = _read_content(message_or_file)
+ if content is None:
+ return False
+
+ # If photo is provided, send photo with chunked caption
+ if photo_path:
+ return _send_photo_with_chunked_caption(bot_token, chat_id, photo_path, content, parse_mode, message_thread_id, disable_web_page_preview, max_retries, retry_delay, delay)
+
+ # If no photo, send text message
+ return _send_chunked_messages(bot_token, chat_id, content, parse_mode, message_thread_id, disable_web_page_preview, max_retries, retry_delay, delay)
+
+
def _send_single_message(bot_token, chat_id, message, parse_mode, message_thread_id, disable_web_page_preview, max_retries, retry_delay):
"""
Send a single message to Telegram channel with retry mechanism.
@@ -127,13 +225,14 @@ def _send_single_message(bot_token, chat_id, message, parse_mode, message_thread
return False
result = response.json()
+ print(f"🔍 Full Telegram API Response: {result}")
+
if result.get('ok'):
thread_info = f" (thread {message_thread_id})" if message_thread_id is not None else ""
print(f"✅ Message sent successfully to chat {chat_id}{thread_info}")
return True
else:
print(f"❌ Telegram API Error: {result.get('description', 'Unknown error')}")
- print(f"❌ Full response: {result}")
if attempt < max_retries:
print(f"âŗ Waiting {retry_delay} seconds before retry...")
continue
@@ -163,6 +262,96 @@ def _send_single_message(bot_token, chat_id, message, parse_mode, message_thread
return False
+def _send_photo(bot_token, chat_id, photo_path, caption, parse_mode, message_thread_id, max_retries, retry_delay):
+ """
+ Send a photo to Telegram channel with retry mechanism.
+
+ Args:
+ bot_token (str): Telegram bot token
+ chat_id (str): Telegram chat ID
+ photo_path (str): Path to photo file
+ caption (str): Photo caption
+ parse_mode (str): Parse mode for caption formatting
+ message_thread_id (int, optional): Thread ID for group messages
+ max_retries (int): Maximum number of retry attempts
+ retry_delay (int): Delay between retry attempts in seconds
+
+ Returns:
+ bool: True if successful, False otherwise
+ """
+ for attempt in range(max_retries + 1):
+ if attempt > 0:
+ print(f"🔄 Retry attempt {attempt}/{max_retries}...")
+ time.sleep(retry_delay)
+
+ url = f"https://api.telegram.org/bot{bot_token}/sendPhoto"
+
+ # Prepare files and data for multipart/form-data
+ with open(photo_path, 'rb') as photo_file:
+ files = {'photo': photo_file}
+
+ data = {
+ 'chat_id': chat_id,
+ 'caption': caption,
+ 'parse_mode': parse_mode
+ }
+
+ # Add thread ID if provided
+ if message_thread_id is not None:
+ data['message_thread_id'] = message_thread_id
+
+ try:
+ response = requests.post(url, files=files, data=data, timeout=60)
+
+ # Always print response for debugging
+ print(f"🔍 Telegram API Response: {response.status_code}")
+
+ if response.status_code != 200:
+ print(f"❌ HTTP Error {response.status_code}: {response.text}")
+ if attempt < max_retries:
+ print(f"âŗ Waiting {retry_delay} seconds before retry...")
+ continue
+ else:
+ return False
+
+ result = response.json()
+ print(f"🔍 Full Telegram API Response: {result}")
+
+ if result.get('ok'):
+ thread_info = f" (thread {message_thread_id})" if message_thread_id is not None else ""
+ print(f"✅ Photo sent successfully to chat {chat_id}{thread_info}")
+ return True
+ else:
+ print(f"❌ Telegram API Error: {result.get('description', 'Unknown error')}")
+ if attempt < max_retries:
+ print(f"âŗ Waiting {retry_delay} seconds before retry...")
+ continue
+ else:
+ return False
+
+ except requests.exceptions.RequestException as e:
+ print(f"❌ Network error: {e}")
+ if attempt < max_retries:
+ print(f"âŗ Waiting {retry_delay} seconds before retry...")
+ continue
+ else:
+ return False
+ except Exception as e:
+ print(f"❌ Unexpected error: {e}")
+ if attempt < max_retries:
+ print(f"âŗ Waiting {retry_delay} seconds before retry...")
+ continue
+ else:
+ return False
+
+ # If all retries failed, print the caption that couldn't be sent
+ print(f"❌ Failed to send photo after {max_retries} retries. Caption content:")
+ print("=" * 80)
+ print(caption)
+ print("=" * 80)
+ return False
+
+
def split_message(message, max_length=4000):
"""
Split long message into chunks.
@@ -211,8 +400,8 @@ def main():
parser.add_argument('--message', required=True, help='Message text or path to file to send')
# Optional arguments
- parser.add_argument('--parse-mode', default='Markdown', choices=['Markdown', 'HTML', 'None'],
- help='Parse mode for message formatting (default: Markdown)')
+ parser.add_argument('--parse-mode', default='MarkdownV2', choices=['Markdown', 'MarkdownV2', 'HTML', 'None'],
+ help='Parse mode for message formatting (default: MarkdownV2)')
parser.add_argument('--delay', type=int, default=1,
help='Delay between message chunks in seconds (default: 1)')
parser.add_argument('--max-length', type=int, default=4000,
@@ -225,6 +414,8 @@ def main():
help='Maximum number of retry attempts for failed messages (default: 5)')
parser.add_argument('--retry-delay', type=int, default=10,
help='Delay between retry attempts in seconds (default: 10)')
+ parser.add_argument('--photo',
+ help='Path to photo file to send (optional)')
args = parser.parse_args()
@@ -250,7 +441,8 @@ def main():
disable_web_page_preview=args.disable_web_page_preview,
max_retries=args.max_retries,
retry_delay=args.retry_delay,
- delay=args.delay
+ delay=args.delay,
+ photo_path=args.photo
)
sys.exit(0 if success else 1)
diff --git a/.github/workflows/create_issues_for_muted_tests.yml b/.github/workflows/create_issues_for_muted_tests.yml
index 192593d9910..35ec820147e 100644
--- a/.github/workflows/create_issues_for_muted_tests.yml
+++ b/.github/workflows/create_issues_for_muted_tests.yml
@@ -41,7 +41,7 @@ jobs:
- name: Install dependencies
run: |
python -m pip install --upgrade pip
- pip install ydb[yc] PyGithub requests
+ pip install ydb[yc] PyGithub requests matplotlib numpy
- name: Setup ydb access
uses: ./.github/actions/setup_ci_ydb_service_account_key_file_credentials
@@ -85,7 +85,9 @@ jobs:
TEAM_CHANNELS: ${{ vars.TG_TEAM_CHANNELS }}
run: |
python .github/scripts/telegram/parse_and_send_team_issues.py \
+ --on-mute-change-update \
--file "${{ steps.create_issues.outputs.created_issues_file }}" \
--bot-token "$TELEGRAM_BOT_TOKEN" \
--team-channels "$TEAM_CHANNELS" \
+ --include-plots \
--delay 2
diff --git a/.github/workflows/weekly_telegram_update.yml b/.github/workflows/weekly_telegram_update.yml
new file mode 100644
index 00000000000..9195f53a012
--- /dev/null
+++ b/.github/workflows/weekly_telegram_update.yml
@@ -0,0 +1,46 @@
+name: Weekly Telegram Update
+
+on:
+ schedule:
+ # Every Monday at 11:00 MSK (08:00 UTC)
+ - cron: "0 8 * * 1"
+ workflow_dispatch:
+
+defaults:
+ run:
+ shell: bash
+
+env:
+ GH_TOKEN: ${{ secrets.YDBOT_TOKEN }}
+
+jobs:
+ weekly-telegram-update:
+ name: Send weekly trend updates to Telegram
+ runs-on: [ self-hosted, auto-provisioned, build-preset-analytic-node]
+
+ steps:
+ - name: Checkout
+ uses: actions/checkout@v4
+ with:
+ ref: main
+
+ - name: Setup ydb access
+ uses: ./.github/actions/setup_ci_ydb_service_account_key_file_credentials
+ with:
+ ci_ydb_service_account_key_file_credentials: ${{ secrets.CI_YDB_SERVICE_ACCOUNT_KEY_FILE_CREDENTIALS }}
+
+ - name: Install dependencies
+ run: |
+ python3 -m pip install --upgrade pip
+ pip install ydb[yc] requests matplotlib numpy
+
+ - name: Send weekly trend updates to Telegram
+ env:
+ TELEGRAM_BOT_TOKEN: ${{ secrets.TELEGRAM_YDBOT_TOKEN }}
+ TEAM_CHANNELS: ${{ vars.TG_TEAM_CHANNELS }}
+ run: |
+ python3 .github/scripts/telegram/parse_and_send_team_issues.py \
+ --period-update week \
+ --bot-token "$TELEGRAM_BOT_TOKEN" \
+ --team-channels "$TEAM_CHANNELS" \
+ --delay 2