Support Call Queue Metrics and Five9 Polling

Questions and Answers

What is the purpose of polling data from Five9 via their API?

For observability and to provide critical Support Call Queue metrics.

Where are the configurations for polling data from Five9 stored in the toast ops repo?

maintaining_infrastructure/roles/smartbox/files/five9.py, env/prod/group_vars/tag_toast_type_datadog_monitor.yml, and env/prod/group_vars/tag_toast_type_controller.yml

Who can provide the password for the “five9_datadog_api” username?

@Patrick Sexton (Deactivated) or anyone on the Infra team with appropriate access, and the CareOPs team

What is the process for making changes to the toast-ops repo?

Open a PR with a relevant approver, and then open an OPX ticket for Infra to review and implement the change. Signup and view all the answers

Why is it necessary to have the appropriate approver for changes to the toast-ops repo?

To ensure that changes are reviewed and approved by relevant teams before implementation. Signup and view all the answers

What is the frequency of polling data from Five9?

Roughly every six minutes Signup and view all the answers

Where can logs be found that are generated as the controller sends requests to the API?

index=system_audit sourcetype=syslog Five9 Signup and view all the answers

What is the typical outcome when running checks in the logs?

Successes Signup and view all the answers

What is the soft-error case that is non-critical and requires no action regarding session timeouts?

Session was closed. Please, login again Signup and view all the answers

What is the impact of the soft-error case on metrics?

No impact, no lost metrics Signup and view all the answers

How many common known failure modes may require active intervention to resolve?

2 Signup and view all the answers

What is one of the failure modes that may require active intervention to resolve?

The user name or password that you provided is incorrect, or the account is locked Signup and view all the answers

What is the file location of the Five9 check in the logs?

/etc/datadog-agent/checks.d/five9.py Signup and view all the answers

What is the purpose of the _get_authenticated_client function in the Five9 check?

To get an authenticated client instance Signup and view all the answers

Study Notes

Five9 Polling and Metrics

Five9 is the support management software used by the Customer Care team to handle inbound requests from restaurants.
Data is polled from Five9 via their API every six minutes using a configuration stored in the maintaining_infrastructure/roles/smartbox/files/five9.py and env/prod/group_vars/tag_toast_type_datadog_monitor.yml files.
The username used for API calls is "five9_datadog_api" and the password can be obtained from @Patrick Sexton or the Infra team.

Updating Configs

To make changes to the toast-ops repo, a PR must be opened with relevant team/person as an approver, and an OPX ticket must be opened for infra to review and implement the change.
No manual deploy is required, but changes to the datadog monitor require manual host updates.

Logging

Logs are generated as the controller sends requests to the API, but responses are not captured.
Logs can be found at index=system_audit sourcetype=syslog Five9.
Successful logs show "Done running check five9" and "Running check five9" messages.
Soft-error cases with session timeouts are non-critical and require no action.

Failure Modes

Two common failure modes may require active intervention to resolve:
- "The user name or password that you provided is incorrect, or the account is locked."
- Session timeouts with the error message "Session was closed.Please, login again."