On-Call Rotation
Each team in the software department has a rotating on-call schedule. This helps to limit the number of potential distractions each person may face during a normal work week. Previously, everyone on the team received a notice when a new critical issue was submitted. This proved to be somewhat inefficient, as the notification would disrupt everyone's current work, and it would often lead to multiple people working on the same issue.
Telegram Bot
We created a Telegram bot (@Hawk_Telegram_bot) that notifies the on-call engineer of incoming urgent issues. The bot is integrated with our monitoring system, so it automatically sends a message to the on-call engineer when an issue is reported.
Initial Setup
New members of the software team are expected to send a /start message to the bot to initiate a conversation. This establishes a "conversation" between you and the bot so the bot is allowed to notify you when issues arise.
Notification Schedule
To protect the on-call engineer's time, notifications of new issues will only be sent on weekdays between the hours of 7:00 AM and 4:00 PM. Issues reported outside of these hours will be sent to the on-call engineer the following workday after they acknowledge they are on call.
Commands
The Telegram bot has two built-in commands:
/startThis command initiates a conversation with the bot. The bot responds with some basic instructions and recommendations for the on-call engineer.
/checkThis command checks to see if there are any issues that have not been "acknowledged". Please note that this command will show you all unacknowledged issues, including those from another department.
Notification Triggers
The following types of issues will trigger a notification to the on-call engineer:
Design automations that are delayed for any reason (automations team only)
New "critical/blocking" issues reported in YouTrack
New Helpdesk tickets (portal team only)
On-Call Responsibilities
There are a few things to know about your responsibilities while you are on call.
Acknowledging Responsibility
You will receive a notification at 7:05 AM asking you to acknowledge that you are on call. This is to ensure that you are available to respond to any issues that may arise during the day.
To confirm, simply tap the "acknowledge" button at the bottom of the message.
If you do not confirm by 7:20 AM, the manager of your team will be notified to verify your availability. If you are unavailable (e.g. taking a sick day), they will adjust the schedule to assign someone else on call.
Responding to Issues
When you receive a notification about a new issue, you are expected to respond in a timely manner.
All new issues include an "acknowledge to view" button at the bottom of the message. You are expected to tap this button to indicate that you have seen the issue and are working on it. If this button is not tapped, the issue will be presented to the on-call person for the next day when they confirm they are on call.
Helpdesk Ticket Workflow (Portal Team)
The portal team has designed a specific workflow for how to respond to helpdesk issues. Click here to view it.