Riva Pro-active Support & Alerts Monitoring PDF

Pro-active Support & Alerts Monitoring The purpose of this manual is to layout a daily process for pro-active support that will enable the Riva Success Team to address issues before they become “incidents” that negatively impact our clients. This will involve reviewing alerts, dashboards, Riva Clou...

Pro-active Support & Alerts Monitoring The purpose of this manual is to layout a daily process for pro-active support that will enable the Riva Success Team to address issues before they become “incidents” that negatively impact our clients. This will involve reviewing alerts, dashboards, Riva Cloud accounts, pod and data center status and making decisions about what the next steps should be. A critical incident is defined as something that affects an entire data center, pod, ICP, or multiple clients. In this case, it is important you gather as much information as possible and follow the Incident Response Procedure – Riva Cloud Incident Response Policy and Riva Cloud Incident Response Procedure. The following items will be covered in this document: 1 Daily Process Checklist 1.1 Change Management 2 Proactive Monitoring Overview 3 Pod Dashboard 4 Proactive Support Dashboard 5 Urgent Priority Alerts 5.1 Alert - Riva Cloud Sync - last AgentRunning by time by pod node 5.2 Alert - Riva Cloud - Enabled load factors 5.3 Alert - Riva Web - redis exceptions 6 High Priority Alerts 6.1 Alert - Riva Cloud Sync - Inactive Policies 6.1.1 Edit user lookup table - usage 6.2 Alert - Riva Cloud - Identify policies with DestinationEntityKeyFilter key enabled 6.3 Alert - Riva Cloud - Identify policies with Sync.Crm. CheckForSyncConfigurationOnFirstSync key 7 Medium Priority Alerts 7.1 Alert - Riva Cloud - Identify policies with recent pod changes and no sync logs in the last 8 hours 7.2 Alert - Riva Cloud - GSC Regatherer - Error adding user to policy 7.3 Alert - Riva Sync - Conflicting entity detection 7.4 Alert - Riva Cloud - Failed automatic renames 8 Low Priority Alerts 8.1 Alert - Riva Cloud - Expiry Date exceed for policy 9 Other Alerts 9.1 Alert - Riva Sync - Cancelled or suspended or expired policies with recent sync 9.2 Alert - Riva Cloud - Credentials and system user issues by policyName 9.3 Alert - Riva Cloud - Sync - Policies with CRM Trace keys 9.4 Alert - Riva Cloud - Sync - Policies with LoggingLevelOverride TRACE or DEBUG Daily Process Checklist 1. Review the Proactive Support Team's channel to ensure there are no "urgent" alerts requiring attention. For example, the "Alert - Riva Cloud Sync - Last AgentRunning time by pod node." 2. Check the Zendesk queue for any open tickets, ensuring that all tickets have been addressed and followed up on. 3. Review the Pod Dashboard and verify that all pod statuses are healthy. 4. Review outstanding Proactive Support alerts and take the appropriate action. Details for each alert can be found below. Change Management No changes should be made to any Riva Cloud account without an accompanying Zendesk ticket. If you're working on an alert and determine that a policy modification is necessary (such as removing a key), you must create a Zendesk ticket to track the change. This ensures accountability and aligns with Riva Cloud's Change Management process. The ticket should outline the changes being made, the reasons for the changes, and provide a link to the relevant alert or Team's thread. If a ticket already exists, you can use it to track the changes. Proactive Monitoring Overview The monitoring system is divided into three main categories: urgent/high priority alerts, medium/low priority alerts, and other alerts. Urgent and High Priority Alerts These alerts are sent to the Proactive Support Alert Team’s channel, Zendesk ticket is created (with suppression) and the Proactive Support Dashboard. The alerts in this category include: Urgent Alerts: Riva Cloud Sync – Last AgentRunning time by pod node Riva Cloud – Enabled load factors Riva Web – Redis exceptions High Priority Alerts: Riva Cloud Sync – Inactive Policies Riva Cloud – Identify policies with DestinationEntityFilter key enabled Riva Cloud – Identify policies with Sync.Crm.CheckForSyncConfigurationOnFirstSync key Medium and Low Priority Alerts These alerts are managed from the Riva Cloud - Proactive Support - Alerts dashboard. Medium Priority Alerts: Riva Cloud – Policies with recent pod changes and no sync logs in the last 8 hours Riva Cloud – GSC Regatherer – Error adding user to policy Riva Sync – Conflicting entity detection Failed automatic renames (once created) Low Priority Alerts: Riva Cloud – Expiry Date exceeded for policy Other Alerts These alerts are not critical to Riva’s uptime or the end user experience but are related to maintenance and necessary clean-up tasks. The alerts in this category include: Riva Sync – Cancelled, suspended, or expired policies with recent sync Riva Cloud – Credentials and system user issues by policy name Riva Cloud – Sync – Policies with CRM Trace keys Riva Cloud – Sync – Policies with LoggingLevelOverride set to TRACE or DEBUG Pod Dashboard When you start your shift you should review the Riva Cloud Sync - Pod Node Policy and User Last sync log times dashboard. This dashboard provides a high-level overview of the pods and their status. If any pod is flagged, or appears to not have synced in a extended period of time it should be investigate to confirm there is not a problem. If you click on the pod, the Riva Cloud Sync - Pod status - real time dashboard will open. This will give you information on the pod, like AgentRunning and last activity. Towards the bottom of the dashboard there is a "Pod Status" panel, which will give you a quick status on the pod: Proactive Support Dashboard The dashboard contains a list of all the alerts (including the urgent and high priority alerts). This alerts should be managed and actioned from this dashboard. You can click on any of the jobs and the alert details will be provided in the "Selected alert/job" panel: Once you have actioned the alert, you can remove the alert from tracking by selecting the "Remove all jobs/results". You will be re-directed to a new page, where you need to confirm the removal: Note, you might be prompted with a security warning. Select "Run Query Anyway". By completing these steps, that alert will be removed from the list. Note, sometimes, when you click on an alert you will get a message that says "Error in 'SearchOperator: loadjob': Cannot find job_id". That suggests that the alert/job is older and might not be relevant any more. Double check the issue in the logs, and if everything is good, remove it from tracking. Urgent Priority Alerts Alerts in this category are deemed critical or have the potential to escalate into critical issues, requiring immediate action. Urgent alerts are directed to the Proactive Support Alerts Teams channel, Zendesk, and the Proactive Support Dashboard. If this happens during regular business hours, it is important that the alert is investigated immediate and escalated to the Cloud Operations team. If this happens during off business hours, you will follow the Incident Response Guide linked above. Alert - Riva Cloud Sync - last AgentRunning by time by pod node Technical Information: This alert runs every five (5) minutes and is triggered if no agent activity is detected for at least twenty (20) minutes. A lack of agent activity indicates that the service for the pod is not running. However, there are scenarios where the alert may trigger, but upon investigation, the pod is found to be functioning as expected. This is often due to a delay in the logging of "Agent Running." The relevant log entry appears as follows: Agent running - version: 3.6.64.25681, total runtime: 1.07:30:26.379619. This log confirms the agent is running, specifies its version, and provides its total uptime. If the alert is triggered, and the logs show that the pod is syncing and the agent is running as expected, no further action is required. Action Steps: If the above alert is triggered, take the following actions. 1. Immediately investigate the pods current status. You can do that by using the Riva Cloud Sync - Pod status - real time dashboard to confirm whether or not the pod is down. That dashboard is explained here. You can also use the Riva Cloud Sync - Pod Node Policy and User last sync log times dashboard to help you determine the current status of the pod and the policies/users. 2. If you determine that the pod is down and users are not syncing, find the Zendesk ticket that would have been created automatically and update it with the following information: a. How long has the pod been down? b. How many users? c. How many policies? d. Is the pod in active business hours? 3. If the outage happens during business hours update the ticket and reach out to the Cloud Operations team to review. It is your responsibility to ensure that a Cloud Operations team member takes responsibility for the ticket and issue. 4. If during off business hours, you will follow the Incident Response Procedure linked above. 5. Once a senior team member (Cloud Operations/On-Call) has assumed responsibility remove the alert from tracking. Note, if the pod is down, but does not have any active policies on it, then you should make note of the issue per the above details in the Zendesk ticket and inform the Cloud Operations team. This specific scenario does not require waking someone up to resolve. On the other hand, if the pod has many active policies, is in business hours, then the Incident Response Procedure should be followed. Alert - Riva Cloud - Enabled load factors Technical Information: Most pods operate in an active/passive setup, meaning one node is active and syncing while the other remains passive. During a failover, the passive node becomes active. A load factor is a configuration setting that determines whether a node is active and can handle user assignments. In an active/passive setup, load factors are typically set to either 1,0 or 0,1, depending on which node is active. Below is an example of the load factor configuration: An alert is configured to trigger if a load factor of 1,1 is detected, as this could indicate a misconfiguration that may assign users to the wrong node. Note that some pods, such as blksp and edup, are configured for active/active operation. Action Steps: If the above alert is triggered, take the following actions: 1. Find the Zendesk ticket that was automatically created. 2. Confirm in the logs whether or not the load factors are set incorrectly - you can do that by searching for the key "GlobalSyncConfiguration.LoadFactors" in Splunk, along with the pod. 3. Confirm if users are syncing on both nodes. 4. If they are syncing on both nodes: a. If during business hours, escalate to Cloud Operations immediately. b. If during off business hours, the Incident Response Procedure should be followed. 5. If the users are not syncing on both nodes, the ticket should be escalated to the Cloud Operations queue to be further investigated. 6. Once a senior team member (Cloud Operations/On-Call) has assumed responsibility remove the alert from tracking. Note, check Zendesk for similar alerts with the same pod to see if the issue has happened before and what action was taken. Alert - Riva Web - redis exceptions Technical Information: Redis is an in-memory data storage platform used at Riva to support both Riva Web and Riva Insight. If this alert is triggered, it may indicate a malfunction in Redis, potentially impacting the functionality of Riva Web or Riva Insight. Action: If the above alert is triggered, take the following actions: 1. Find the Zendesk ticket that was automatically created. 2. Determine the severity: a.  2. a. If the alert is triggered, but the error has only happened once, escalate the ticket to the Cloud Operations team for review. b. If the alert is triggered and there are a lot of Redis errors, it could indicate bigger problem. This needs to be escalated immediately either to Cloud Operations (during business hours) or by following the Incident Response Procedure, if after regular business hours. 3. Once a senior team member (Cloud Operations/On-Call) has assumed responsibility remove the alert from tracking. Note, check Zendesk to see if any tickets with the same exception before escalating. It is possible a ticket already exists and is being addressed. If that is the case, you can merge the ticket. High Priority Alerts High-priority alerts are considered highly important and may affect the end-user experience. These alerts are sent to the Proactive Support Alerts Teams channel and the Proactive Support Dashboard. When a high-priority alert is triggered, it should be promptly investigated and escalated to the Cloud Operations team for further review and resolution. Alert - Riva Cloud Sync - Inactive Policies Inactive policy tracking in Splunk is managed through multiple components: 1. Reporting System: Tracks the last log time for each policy/user. 2. Scheduled Alert: Monitors policies based on the following criteria: User count 50: Last log was more than 8 hours ago. User count < 50: Last log was more than 2 days ago. Policies meeting these criteria are reported via email (details below). 3. Dashboard Tool: Allows manual removal of policies from tracking. The reporting system automatically tracks new policies and resumes tracking for previously inactive policies that become active again. The scheduled alert monitors these tracked policies for inactivity, while the dashboard facilitates investigation and removal from tracking when necessary. Dashboard Tool: The dashboard, “TOOL - Edit user lookup table,” is used to remove policies from tracking when necessary (e.g., expired trials, cancelled accounts, NFR, demo policies). Links to the dashboard for each data center are provided in the alert email: US: Splunk Dashboard Link EU: Splunk Dashboard Link CA: Splunk Dashboard Link AU: Splunk Dashboard Link The tracking system continues to monitor policy activity and will resume tracking policies that become active after being removed from tracking via the dashboard. Action Steps 1. Monitor Alert Emails Regularly: Ensure all alert emails are reviewed promptly. Watch for larger policies becoming inactive unexpectedly and act quickly to restore sync if necessary. Monitor all other policies to determine if sync restoration is required. 2. Investigate Policies Using the Dashboard: Use the dashboard links provided in the alert email to investigate inactive policies. For policies identified as inactive: Restore sync immediately if applicable. Remove from tracking if there is a valid reason (e.g., expired trial, cancelled account). 3. Take Action on Alerts: If the cause of inactivity is unclear after investigation, create a Zendesk ticket for further review. If sync restoration is completed or the policy is removed from tracking, update the alert in the team channel with the actions taken. 4. Frequent Monitoring and Cleanup: Proactive team members should regularly monitor and clean up the list of inactive policies to simplify future tracking efforts.  4. The team member assigned the proactive role is responsible for maintaining this process. 5. Training and Additional Support: Training is available as needed (contact Julian, Christian, or David). Follow the instructions provided to ensure accurate handling of inactive policies. Edit user lookup table - usage policyName – click on a policyName to remove it from tracking (this is the first stage of removal). However, do not remove any policies from tracking prior to researching and determining the exact reason why the policy has become inactive. Users – click on a value in the “users” column to open the “Riva Cloud Sync - Sync times activities and policy changes” dashboard. This dashboard can quickly show information such as recent activity for a policy, changes to policy settings and advance key settings, and recent policy disables (if they can be located). Domain – Open a Zendesk search on the domain – use this if no relevant results can be found when searching Zendesk for the admin. rcAdmin - Open a Zendesk search for the admin, tickets should be ordered with the most recent edit first. If the user is a company user, and in some other circumstances, the domain search (above) might have to be used to find any recent tickets. timeSinceLastActivityLog – click on a value in this column to open the dashboard “Riva Cloud Sync - recent activities by policy”, which reveals recent sync activity logs for the selected policy. If you’re unable to determine what happened using the above resources, copy the value from the rcAdmin column and paste it into a Riva Cloud search to review account details on Riva Cloud. Use all the above resources to determine the “reason” the policy is inactive; if you are unable to determine the reason at this point, seek assistance from Ops and anyone that might’ve been recently working with the client. Also look to involve CX at this point, especially if client contact is required. If you’re able to determine that the policy should not be inactive, take all steps necessary to resolve and /or notify those that should be involved. This could range from CX/the admin, through to Ops/CET and others, depending on the size of the account and findings. If you’re able to determine that the policy should be inactive, with an obvious reason why, and should no longer be tracked, go ahead, and remove it from tracking (with care) using the following steps. The current list of “reasons”: Expired trial Cancelled UAT (but triple-check to ensure that the policy should be inactive) NFR Non-syncing admin Moved or migrated Disabled by client, 1-4 Users (message CX to correspond with client) Open ticket (be careful in this circumstance, clients occasionally go unresponsive, and tickets can be closed unresolved, when we really should track all the way through to confirmed resolution). Future cancellation, policy disabled by client (work with CX and review RC settings to ensure that the policy should remain inactive/disabled until the cancellation date). Note, we have implemented a comment section that is paired with the “reason” picklist. This will allow you to add additional comments on the reason you removed the policy from the lookup table. This is not required; however, it is important you add additional context where required. If the policy already has an open Zendesk ticket and you remove the policy from the lookup table for that reason, please provide a link to the Zendesk ticket in the “comment” field: To remove an inactive policy from tracking Click on the policyName to select it (panel “Policies with no recent sync…”) In the 2nd panel (“Click on the policyName below to permanently remove all tracked users from the lookup table…”), select the Reason for removal, using the dropdown input Confirm that the correct policyName is listed in the results section in the second panel, to continue the removal process, click on the policy name in this results section. Review the image directly below the results sections (panel #2), details are here on how to work around any security warnings presented by Splunk There are two panels, below, that require you to click on the warning triangle, then click “Run Query Anyway”, as described in the image. Ensure that the steps are followed for both panels If successful, the following panels will be updated: “Edit history with revision” “Updated user list…” Each time after a policy has been removed from tracking: Reload the dashboard Confirm the policy is removed from the tracking list (panel #1) Confirm the policy removal is logged in the edit history (“Edit history” panel) Alert - Riva Cloud - Identify policies with DestinationEntityKeyFilter key enabled Technical Details: This alert identifies any policy where the key Sync.Crm.DestinationEntityFilter is enabled. It provides details including the pod, admin, primaryEntityName, and a link to a related search in Zendesk. Action Steps: 1. Review the Policy: Check the Riva Cloud account to confirm if the key is present and enabled on the policy. 2. If the Key is Disabled: Remove the key from the policy. Refer to the Change Management section before making any changes. 3. If the Key is Enabled: Review Zendesk for any open tickets related to the client. If an open ticket exists: Add an internal note stating that a destination entity filter is in place. Contact the team member responsible to confirm whether the filter should remain active. If the filter is not required, remove it. 4. If No Open Tickets Exist: Remove the key from the policy.  4. Ensure users resume syncing as expected. 5. Finalize the Process: When completed, remove the alert from tracking on the Proactive Support Dashboard Alert - Riva Cloud - Identify policies with Sync.Crm. CheckForSyncConfigurationOnFirstSync key Technical Details: This alert identifies policies where the Sync.Crm.CheckForSyncConfigurationOnFirstSync key is present. The alert table provides the pod, admin, and a link to a Zendesk search. Action Steps: 1. Review the Policy: Check the Riva Cloud account to confirm if the key is present on the policy. This key should not be active on accounts currently syncing. 2. If the Key is Present: Remove the key from the policy. Before proceeding, refer to the Change Management section. Note: There may be valid reasons for this key to remain on a policy. If you are uncertain about its purpose, create a ticket for the Riva Cloud Operations team to review before making changes. 3. Check Zendesk for Relevant Tickets: If related tickets exist, add an internal note explaining that the key was removed and provide the reason. 4. If You Suspect the Key Is Needed: Contact the team member responsible to confirm its purpose. Use “@agentName” in the alert reply to request clarification. 5. Finalize the Process: Once resolved, remove the alert from the Proactive Support Dashboard. Medium Priority Alerts High-priority alerts may affect the end-user experience or could cause delays in sync. These alerts are sent to the Proactive Support Dashboard. Alert - Riva Cloud - Identify policies with recent pod changes and no sync logs in the last 8 hours Technical Details: This alert is triggered when a change has occurred, and there are no sync logs within the last eight (8) hours. While this is often intentional (e.g., a policy was intentionally disabled), it may sometimes indicate a misconfiguration preventing the policy from syncing. Action Steps: 1. Ignore Internal or Demo Policies: These do not require action. 2. Investigate Customer Production Policies: If the alert involves a customer production policy, confirm whether the policy is not syncing. 3. Check Riva Cloud: Verify if the policy is disabled. If disabled, determine whether this was intentional. Policies are sometimes left disabled accidentally. 4. Engage with the Customer if Necessary: If unsure whether the policy should remain disabled, create a ticket and contact the customer for confirmation. 5. Escalate if Required: If the cause remains unclear, escalate the ticket to the Cloud Operations team for further investigation. 6. Resolve and Update:  6. Once the issue is resolved or handed off to the Cloud Operations team, remove the alert from the Proactive Support Dashboard. Alert - Riva Cloud - GSC Regatherer - Error adding user to policy Technical Information: This alert indicates that the GSC service is unable to load the user into the policy, preventing them from syncing. Since this issue occurs at the GSC level, it will not be reflected in the user logs. In Riva Cloud, the user will appear as enabled but not syncing. Common causes include corrupted entity.settings files or invalid characters. Action Steps: 1. Create a Zendesk Ticket: Assign the ticket to Riva Cloud Operations and set the priority to “High.” Include the following details: Pod Admin File paths Affected user(s) Reference ID for the GSC cycle (available in the alert). 2. Operations Team Resolution: The Riva Cloud Operations team will review the information and resolve the GSC error for the affected user(s). 3. Update the Proactive Support Dashboard: Once the issue is resolved or escalated to the Cloud Operations team, remove the alert from the Proactive Support Dashboard. Alert - Riva Sync - Conflicting entity detection Technical Details: This alert identifies users with the message “Conflicting entity detected” in the logs. This indicates that the user was syncing on one policy but is now attempting to sync on a different policy. To prevent data integrity issues, Riva blocks this behavior. Action Steps: 1. Check Zendesk for Open Tickets: Look for any existing tickets related to this user. 2. Verify the Two Policies: If no ticket exists, review the two policies involved. Create a Zendesk ticket and contact the client to determine why the user is attempting to sync on two policies. Common scenarios include: The user was moved between UAT and production. The user previously synced on a Riva On-Premise policy. 3. Important Notes: Users cannot be moved between policies without involvement from Riva Cloud Operations. Users cannot be duplicated in Riva Cloud unless using a z-user or with assistance from Riva Cloud Operations. There is likely an existing ticket explaining what happened with the user. 4. Update and Resolve: Once the issue is resolved or escalated to the Cloud Operations team, remove the alert from the Proactive Support Dashboard. Alert - Riva Cloud - Failed automatic renames Technical Details: For customers using an external user gatherer (CrmGather/MailGather), Riva automatically renames users in Riva Cloud when their username changes at the source. However, certain scenarios prevent Riva from completing the renaming process. This alert is triggered when such a scenario occurs. Action Steps: 1. Review the Alert:  1. Analyze the details of the alert. 2. Create a Zendesk Ticket: Group renames with the same domain/policy into a single ticket (do not create individual tickets for each user). Include the following information in the ticket: Pod Admin Affected users Error details 3. Update and Resolve: Once the issue is resolved or escalated to the Cloud Operations team, remove the alert from the Proactive Support Dashboard. Low Priority Alerts Low priority alerts do not have an immediate impact on the user experience but still require investigation and resolution. Alert - Riva Cloud - Expiry Date exceed for policy Technical Details: This alert identifies policies with the log message "Expiry Date exceeded for policy." If this message is present, it means one of the following: The policy is expired but still attempting to sync. The policy was previously expired but has since been extended or assigned a subscription and is now attempting to sync. The presence of the Sync.Crm.PolicyExpirationDate key will prevent syncing. Action Steps: 1. Check the Riva Cloud Account: Determine if the account is expired. 2. If the Account is Expired: Ensure the policy is disabled. If an open ticket exists, add an internal note to inform the team member that the account is expired and the policy has been disabled. 3. If the Account was Extended: Confirm that the Sync.Crm.PolicyExpirationDate key reflects the updated expiry date. 4. If a Subscription was Applied After Expiration: Ensure the Sync.Crm.PolicyExpirationDate key is removed from the policy. Verify that syncing resumes as expected. 5. Update the Alert Reply: Once the issue is resolved or escalated to the Cloud Operations team, remove the alert from the Proactive Support Dashboard Other Alerts Several additional alerts fall under low priority or maintenance categories. These should be reviewed periodically. Alert - Riva Sync - Cancelled or suspended or expired policies with recent sync Technical Details: This alert identifies cancelled, suspended, or expired policies that still have recent activity logs, indicating they are currently syncing. Action Steps: 1. Verify Policy Status: Check the policies in Riva Cloud to confirm if they are cancelled, suspended, or expired. If they are, ensure the policy is disabled. 2.  2. Disable the Policy if Needed: If the policy is not already disabled, proceed to disable it. 3. Check for Open Tickets: Review Zendesk for any open tickets (e.g., GSS). If a ticket exists, disable the policy as needed and add an internal note explaining why the policy was disabled. 4. Check for Active Subscriptions: In cases where a policy appears in the alert but has a recently applied subscription, confirm the policy is enabled and syncing correctly. Alert - Riva Cloud - Credentials and system user issues by policyName Technical Details This alert identifies connection or user errors that prevent successful connections to the target system, such as impersonation issues, primary SMTP errors, server errors, and others. The alert is sent as individual emails, one per policy, to help throttle alerts by policyName and simplify copying details into the ticket. Action Steps: 1. Review Zendesk for Open Tickets: Check if there are any open tickets related to this issue. 2. Create a Zendesk Ticket (if none exists): If no tickets are found, create a new ticket for the Success team to follow up. Include the following details in the ticket: Riva Cloud account Pod Errors Affected users 3. Add Ticket Tag: Add the "agent_first_reply" tag to the ticket. Alert - Riva Cloud - Sync - Policies with CRM Trace keys Technical Details: This alert detects policies containing the following key: Crm.Trace.ToCrmObject.[module]. Its purpose is to identify unnecessary excessive logging—these keys should not remain on policies indefinitely. Action Steps: 1. Check Zendesk for Open Tickets: Verify if there are any open tickets where an investigation is ongoing. If no tickets are found, proceed to remove the key from the policy. 2. If an Investigation is Ongoing: If an open ticket is found, add an internal note to remind the team member to remove the key once the investigation is complete. Alert - Riva Cloud - Sync - Policies with LoggingLevelOverride TRACE or DEBUG Technical Details: This alert identifies policies where the logging level is set to TRACE or DEBUG. Its purpose is to detect unnecessary excessive logging that should not be enabled. Action Steps: 1. Check Zendesk for Open Tickets: Verify if there are any open tickets related to an ongoing investigation. If no tickets are found, proceed to remove the key from the policy. Note: Before removing the key, please refer to the Change Management section. 2. If an Investigation is Ongoing: 2. If an open ticket is found, add an internal note to remind the team member to remove the key once the investigation is complete.