20-PAM-ADMIN-Troubleshooting.pdf

PAM Administration Troubleshooting © 2023 CyberArk Software Ltd. All rights reserved  Agenda By the end of this session, you will be able to: 1. Describe...

PAM Administration Troubleshooting © 2023 CyberArk Software Ltd. All rights reserved  Agenda By the end of this session, you will be able to: 1. Describe the basic flow for troubleshooting issues in the CyberArk environment 2. Describe, locate, and manage the log files generated by the Vault and various components 3. Describe, configure and use the xRay agent © 2023 CyberArk Software Ltd. All rights reserved  Troubleshooting Flow © 2023 CyberArk Software Ltd. All rights reserved  Overview The basic troubleshooting methodology for the PAM solution requires a thorough understanding of: 1. Your system implementation 2. How components communicate with each other in your environment 3. What is the current behavior compared to the expected behavior? This methodology is designed to provide guidance and might not apply to every scenario It is important to write down any information gathered during this process and any tests performed, as all of this information will be required when opening a case with CyberArk support © 2023 CyberArk Software Ltd. All rights reserved  1. Knowledge of the environment layout 2. Access to the different servers 3. Access to CyberArk Knowledgebase Prerequisites (Customer Community) 4. Access to CyberArk documentation (publicly available online) The latest version of the documentation will contain the most recent enhancements and notes. © 2023 CyberArk Software Ltd. All rights reserved  Troubleshooting Flow Isolate the issue Understand the Initial questions, to specific Check relevant Follow-up environment’s focus on user scenario by trying logs questions topology experience to reproduce Check documentation Here is an overview of the basic steps and of the troubleshooting flow knowledgebase During this presentation we will review each block on the flow and apply it to a basic scenario Contact Support © 2023 CyberArk Software Ltd. All rights reserved  Understanding the Environment Isolate the issue Understand the Initial questions, to specific Check relevant Follow-up environment’s focus on user scenario by trying logs questions topology experience to reproduce Check documentation and Which components are installed and where? knowledgebase What is the version of the relevant components? Is a Load Balancer being used? Contact Support Are DR or HA solutions implemented? © 2023 CyberArk Software Ltd. All rights reserved  Initial Questions Isolate the issue Understand the Initial questions, to specific Check relevant Follow-up environment’s focus on user scenario by trying logs questions topology experience to reproduce User experience? Check documentation Affected users? and Error message displayed? knowledgebase New implementation or worked and broken? Something changed when this issue started? Contact Was there a process crash? Support How does it impact production? Reproducible? © 2023 CyberArk Software Ltd. All rights reserved  Isolation and Reproduction Isolate the issue Understand the Initial questions, to specific Check relevant Follow-up environment’s focus on user scenario by trying logs questions topology experience to reproduce Reproducible Check documentation Modify a variable and try to reproduce again. and Repeat in different scenarios knowledgebase Write down each scenario and the outcome of the test Review the logs of reproduced scenarios (working and not working) Contact Not reproducible Support Review the logs relevant for the reported flow © 2023 CyberArk Software Ltd. All rights reserved  Checking the Logs Isolate the issue Understand the Initial questions, to specific Check relevant Follow-up environment’s focus on user scenario by trying logs questions topology experience to reproduce Check documentation and Log location knowledgebase Log types Log correlation Contact Support © 2023 CyberArk Software Ltd. All rights reserved  Follow-Up Questions Isolate the issue Understand the Initial questions, to specific Check relevant Follow-up environment’s focus on user scenario by trying logs questions topology experience to reproduce Check documentation and knowledgebase Review and refine questions Contact Support © 2023 CyberArk Software Ltd. All rights reserved  Documentations and Knowledge Base Isolate the issue Understand the Initial questions, to specific Check relevant Follow-up environment’s focus on user scenario by trying logs questions topology experience to reproduce Check Colleagues and end users documentation and Knowledge base knowledgebase Messages and Responses document; Installation and implementation Contact documents Support Re-run scenarios © 2023 CyberArk Software Ltd. All rights reserved  Contacting CyberArk Support Isolate the issue Understand the Initial questions, to specific Check relevant Follow-up environment’s focus on user scenario by trying logs questions topology experience to reproduce Environment details? Check User experience? documentation and Did it work in the past? knowledgebase Are there any error messages? Flow, current and expected behavior? Troubleshooting steps? Contact Support Steps to reproduce this issue? All relevant logs, screenshots and configuration files © 2023 CyberArk Software Ltd. All rights reserved  Troubleshooting Flow: Example © 2023 CyberArk Software Ltd. All rights reserved  User Unable to Login A user is unable to login to the PrivateArk client using the administrator user. They see the following message. © 2023 CyberArk Software Ltd. All rights reserved  User Unable to Login Understand the Environment Isolate the issue Understand the Initial questions, to specific Check relevant Follow-up environment’s focus on user scenario by trying logs questions topology experience to reproduce Check documentation and knowledgebase 1 Vault Prod 1 DR vault 1 PVWA, CPM, PSM Contact All running with Version 12.6 on Windows 2019 servers Support © 2023 CyberArk Software Ltd. All rights reserved  User Unable to Login Initial Questions Isolate the issue Understand the Initial questions, to specific Check relevant Follow-up environment’s focus on user scenario by trying logs questions topology experience to reproduce Check documentation and Is this issue experienced by all users? One user knowledgebase Did it work before? Yes Was something changed? No Contact Support Is there any error message? Yes © 2023 CyberArk Software Ltd. All rights reserved  User Unable to Login Isolation and Reproduction Isolate the issue Understand the Initial questions, to specific Check relevant Follow-up environment’s focus on user scenario by trying logs questions topology experience to reproduce Check documentation and knowledgebase Same issue via PVWA? Yes Reproducible? Yes Contact Support © 2023 CyberArk Software Ltd. All rights reserved  User Unable to Login Checking the Logs Isolate the issue Understand the Initial questions, to specific Check relevant Follow-up environment’s focus on user scenario by trying logs questions topology experience to reproduce Check documentation and knowledgebase Error origin ITA Origin is vault ITATS004E Contact Support Vault logs: ITAlog.log © 2023 CyberArk Software Ltd. All rights reserved Trace.d0  Check Messages and Responses Try to identify the problem by searching in the Messages and Responses page in on the online documentation © 2023 CyberArk Software Ltd. All rights reserved  Check Messages and Responses Messages displayed to end users are intentionally generic, listing many possible causes. © 2023 CyberArk Software Ltd. All rights reserved  Check Messages and Responses Because the error message starts with ITA, we know that the Vault server originated this error. At this point we will go to the Vault server and inspect the ITA log. There may be multiple log entries for the same problem. Try to find the first entry related to this problem When looking at the ITA log, we see an error message ITATS528E with a code of 66 When we search for that error, we see the exact cause of the problem and the solution. © 2023 CyberArk Software Ltd. All rights reserved  User Unable to Login Solution Isolate the issue Understand the Initial questions, to specific Check relevant Follow-up environment’s focus on user scenario by trying logs questions topology experience to reproduce Check documentation and knowledgebase Does resetting the user password solve the problem? Yes (solved) Contact Support No © 2023 CyberArk Software Ltd. All rights reserved  User Unable to Login Problem Not Resolved Isolate the issue Understand the Initial questions, to specific Check relevant Follow-up environment’s focus on user scenario by trying logs questions topology experience to reproduce Check documentation and knowledgebase In the event of another login failure: Check the relevant logs again – same error or a new one? Repeat the troubleshooting flow Contact Contact support when no more logical steps are found Support © 2023 CyberArk Software Ltd. All rights reserved  Logs In this section we will discuss the logs generated by the various system components, how to set the debug mode, and the logs location © 2023 CyberArk Software Ltd. All rights reserved  Overview © 2023 CyberArk Software Ltd. All rights reserved  Types of Logs Log files are divided into several types: Console Provides component-level entries Trace Provides detailed entries of workflows Log such as service up or down Log related to that component Error Exists in some components, Debug Those logs may come in different types, Log and will include only error entries Log sometimes they will be the trace files, with additional information and sometimes they will come at a form of separate files depending on the component. For the full list of log locations, please see the implementation guide © 2023 CyberArk Software Ltd. All rights reserved  Understanding CyberArk Logs The log message code is built from four segments for example: ITA – The source component of the message is the Vault server ITA FW 001 I FW – The module with the message is the Vault FW Firewall is open for client communication 001 – Message number I – The message category Log messages are separated into four major categories: Informational: Error: ITAFW001I Firewall is open for client communication ITATS691E LDAP synchronization error Warning: System: ITATS319W Firewall contains external rules ITADB367S Server unable to communicate with firewall See CyberArk Messages and Responses for additional information © 2023 CyberArk Software Ltd. All rights reserved  Reviewing the Logs Once we get to a point where we need to go over log files, there are a number of questions to ask: Which log file do we need to review? What do we search for? ⎼ Keywords (Error, Failed, Failure…) ⎼ Timestamps ⎼ User name ⎼ Object name (Account name, safe name) Are there correlated entries in other logs? ⎼ Log events and time of the issue ⎼ Different components ⎼ CyberArk logs and OS logs © 2023 CyberArk Software Ltd. All rights reserved  Debug Mode and Log Location © 2023 CyberArk Software Ltd. All rights reserved  Set the Debug Mode for the Vault ITAlog The Vault debug levels can be changed in the dbparmi.ini file (requires a restart) The Vault debug levels can be changed without a restart using the PARclient or Central Administration Station © 2023 CyberArk Software Ltd. All rights reserved  Set the Debug Mode for the Components Debug mode for components can be set in the configuration files stored on the Vault or via the PVWA Web UI Set the debug level for CPM Set the debug level for PSM © 2023 CyberArk Software Ltd. All rights reserved  Log Locations and Configuring the Debug Levels Detailed information about setting debug level for different components and location of the log files can be found in the online documentation Setting Vault log levels to Debug should only be done under the guidance of CyberArk Support © 2023 CyberArk Software Ltd. All rights reserved Cheat Sheet – Vault and Related Components Vault Changes Require a Vault Restart Ene Event Notification Engine Configuration Configuration \Program Files\PrivateArk\Server\Event Notification DBParm.ini File Engine\ENEConf.ini File …\Database\my.ini. - Database Configuration File Vault ➔Safe:”Notification Engine”➔root\EventNotificationEngine.ini Debug EventNotificationEngine.ini Debug DebugLevel=PE(1),PERF(1) - Detailed Vault services debug [Debug] ControllerDebugLevel=1,2,3,4 LDAP(14,15) - Detailed LDAP debug CollectorDebugLevel=1,2 ParserDebugLevel=1,2 Logs Italog.log SMTPSenderDebugLevel=1,2 Disaster Recovery ConfigurationManagerDebugLevel=1,2 Trace.dX (X is a number from 0 to 4) Configuration File PADR.ini Logs ProgramFiles\PrivateArk\Server\EventNotification Engine\Logs\ENEConsole.log …\Database\VaultDB.log - Database log Debug EnableTrace=yes ProgramFiles\PrivateArk\Server\EventNotification Engine\Logs\ENETrace.log Logs PADR.log Logic Container File Name LogicContainer.Log Client Run –PAInfo.exe C:\ProgramFiles Debug Logs In the Client: (x86)\PrivateArk\Server|LogicContainer\LogicContainer.log Tools ➔ Options ➔ Advanced ➔ Log Configuration PAReplicate Backup and Restore Logs (Win XP \Documents and Settings\\Application Data\CyberArk\PrivateArk\PALog.txt Debug In the PAReplicate.exe command executed, add the and Win 2003 following flag: /EnableTrace Logs (Win7 and \Users\\AppData\Roaming\CyberArk\PrivateArk Win 2008 Logs PAReplicate.log © 2023 CyberArk Software Ltd. All rights reserved CLICK “NEXT” TO CONTINUE Cheat Sheet – Components PSM Privileged Session Manager Configuration \Program Files\CyberArk\PSM\Basic_psm.ini File PVWA ➔ Administration Tab ➔ Options ➔ Privileged Session Management Debug PVWA ➔ System tab ➔ Options ➔ Privileged Session Management ➔ CPM Central Password Manager General Settings Configuration Vault ➔ Safe “Password Manager”➔ root\policies\.ini Server Settings ➔ TraceLevels=1,2,3,4,5,6,7 File Recorder settings ➔ TraceLevels=1,2 Debug PVWA ➔ Administration Tab ➔ CPM settings Connection Client Settings ➔ TraceLevels=1,2 Logs \Logs (and subfolders) or according to parameter CPMDebugLevels=2 (default) “LogsFolder” (located in Basic_psm.ini file) 0 – No messages will be written to the trace log. 1 – CPM exceptions will be written to the trace log (Default Level) 2 – CPM trace messages will be written to the trace log. PVWA Password Vault Web Access 3 – CPM CASOS activities will be written to the trace log. Configuration 4 – CPM CASOS debug activities will be written to the trace log. \wwwroot\PasswordVault\web.config File 5 – CPM CASOS errors will be written to the trace log. 6 – All CPM CASOS activities and errors will be written to the trace log. Vault ➔ Safe “PVWAConfig” ➔ root\PVConfiguration.xml Vault ➔ Safe “PVWAConfig” ➔ root\Policies.xml Debug PVWA ➔ Administration Tab ➔Options ➔ Logging Logs – CPM \Program Files\CyberArk\PasswordManager\Logs\pm.log \Program Files\CyberArk\PasswordManager\Logs\pm-error.log\Program DebugLevel=High (options are None/High/Low/Profiling) Files\CyberArk\PasswordManager\Logs\PMConsole.log\Program InformationLevel=High (options are None/High/Low/Profiling) Files\CyberArk\PasswordManager\Logs\PMTrace.log Logs %windir%\temp\ Logs –Plug-ins \Program Files\CyberArk|passwordManager\Logs\ThirdParty\*.log CyberArk.Webapplication.log CyberArk.WebConsole.log CyberArk.WebSession..log © 2023 CyberArk Software Ltd. All rights reserved CLICK “NEXT” TO CONTINUE  In this section we will discuss the CyberArk xRay utility, which can be xRay Agent used to collect log and configuration files from the CyberArk components and share them with CyberArk or partner support © 2023 CyberArk Software Ltd. All rights reserved  Overview CyberArk xRay collects logs and configuration files from PAM components in a simple, single- step process The utility can be run from a remote machine or on any of the CyberArk servers All data files are encrypted during collection, regardless of whether they are collected locally or remotely, and then transferred back to the xRay machine You can share the collected data with your partner or CyberArk, knowing that it is safely encrypted during transfer The utility can be downloaded When sharing with CyberArk, shared data is from the CyberArk Marketplace linked to a case to allow Enterprise Support easy and secure access to the collected data © 2023 CyberArk Software Ltd. All rights reserved  Agent Setup Select the component Select time frame for the collection and collection level. Select Collection scope ⎼ Logs from OS and the application ⎼ Logs from application only Optionally, enable and provide the Active Vault IP address and Administrative user credentials for configuration files collection Agree to the Terms of Use and click Start Collection © 2023 CyberArk Software Ltd. All rights reserved  Monitor Collection Process You can monitor the collection process as it collects the component files © 2023 CyberArk Software Ltd. All rights reserved  Share the Collected Data Once the process is complete, you can select whether to: ⎼ Share the collected data with your Partner ⎼ Share the collected data with CyberArk You can also preview the data before sending When sharing information with CyberArk, make sure you have: ⎼ A Technical Community account ⎼ Case number © 2023 CyberArk Software Ltd. All rights reserved  Documentation Additional information can be found in the CyberArk documentation © 2023 CyberArk Software Ltd. All rights reserved  Summary © 2023 CyberArk Software Ltd. All rights reserved  In this session we covered: The basic flow for troubleshooting issues in the CyberArk environment Summary How locate and manage the log files generated by the Vault and various components How to configure and use the xRay agent © 2023 CyberArk Software Ltd. All rights reserved  Utilities xRay (login required) Additional Community Resources Resources CyberArk Customer Community (login required) CyberArk Subreddit Note: The CyberArk subreddit is not hosted or moderated by CyberArk. Online Training Working with CyberArk Support © 2023 CyberArk Software Ltd. All rights reserved