comp3521_U5_

• Data reliability refers to the accuracy and completeness of computer-processed data, based on intended uses.
• Computer-processed data includes data entered into a computer system or resulting from computer processing.

Understanding Data Reliability

• Computer-processed data examples include data extracts from databases, data warehouses, or data repositories.
• Other examples of compuer-processed data include data maintained in Microsoft Excel or Access.
• Also data extracts from enterprise software applications, public use data, data from web portals, and data summarized in documents.
• Reliability ensures data is reasonably complete, accurate, meets intended purposes, and is not inappropriately altered.
• Completeness means relevant records are present, and fields in each record are populated.
• Accuracy means recorded data reflects the actual underlying information.
• Consistency, a subcategory of accuracy, means data is clear and yields similar results.
• Validity (as used here) refers to whether the data actually represent what is being measured.
• Assessments of reliability consider the research project's characteristics and the risk of using unreliable data.
• Acceptable errors mean risks have been assessed, and errors will not cause doubt of findings, conclusions, or recommendations.

Deciding Whether a Data Reliability Assessment Is Necessary

• To decide if a data reliability assessment is needed, consider the planned use of data
• If data will materially support findings, conclusions, or recommendations, the analyst should assess reliability.
• If findings include only a condition description, assessment is still required.
• Data can include record-level, summary, aggregate data, and estimates/projections based on computer-processed data.
• The audit plan should discuss the plan to assess data reliability and any possible limitations.
• If using data in a report does not significantly impact findings, conclusions, or recommendations, the analyst need not assess reliability; examples include background, context, or examples.
• Such data should be from the best available sources.
• Data providing context that incites the audit or is likely to be scrutinized requires assessment.
• Information used for widely accepted purposes from generally recognized sources, does not need verification procedures.
• Economic statistics from government agencies and authoritative organizations are examples.
• Using this data without assessment needs professional judgment and knowledge of the information's nature.
• Financial audits should follow the Financial Audit Manual and the Federal Information System Controls Audit Manual instead of this guide.
• An information system review assesses and tests all controls in a computer system.
  • Examines general and application controls of a computer system
  • Test whether those controls are being complied with
  • Test data produced by the system
• Information technology specialists can help design an appropriate information system review

Determining the Extent of the Assessment

• The ultimate goal of assessment is whether the data can answer research questions.
• Assessments should focus on relevant portions of the data.
• The extent of an assessment depends on
  • the expected importance of the data
  • strength or weakness of corroborating evidence, and
  • anticipated level of risk in using the data.
• The extent to which corroborating evidence exists and will independently support the findings.
• Corroborating evidence is unique to each review, and its strength or persuasiveness varies.
• Consider the extent to which the corroborating evidence is:
  • Consistent with Yellow Book standards of evidence-sufficiency and appropriateness.
  • Able to provide crucial support.
  • Drawn from multiple sources
  • Drawn from multiple types of evidence.
  • Independent of other sources.
• Risk is the likelihood that using data of questionable reliability could have substantial negative consequences on policy makers' decisions and others.

Planning a Data Reliability Assessment

• When planning a data reliability assessment, decide on the timing, how to document both the plans and the assessment.
• Assessments also consider how obtaining data at summary or record levels will affect report use and data reliability assessment depth.
• The assessment is performed as early as possible on a project, ideally in the design phase.
• The audit plan should include data reliability issues and additional steps.
• A team must take initial steps to test the data and review existing data/system information before making the audit plan final.
• The team should examining data so the team might determine whether data is appropriate for answering the research question.
• Record-level data give the greatest opportunity to analyze the data and fully assess reliability.
• Summary-level data or data subsets require reliability assessment.
• Any process used for summarizing or extracting data should be understood.
• Obtaining the record derivation code allows you to check the correct criteria in records for decreasing the chance of missing records.
• All work performed as part of the assessment must be documented.
• Required documentation includes assessment steps plan, test results, documentation reviews, and interviews related to data reliability.
• Decisions made during the assessment, including the final reliability determination, should be summarized.

Steps in the Assessment

• Data reliability is a process including reviewing existing information, interviews, advanced electronic analysis, source document tracing and reviewing selected system controls.
• Deciding which steps to take is an iterative process.
• Start with reviewing information and basic testing, and then take further steps for gathering information.
• Mix of steps depends on potential weaknesses identified and circumstances, such as importance to the review and corroborating evidence.
• Focus on aspects of the data that pose the greatest risk.
• As discussed in section 5, these steps take place early in the project and include the audit team members, as well as appropriate technical staff.
• A review of existing information helps determine what's already known about the data and processing.
• Collecting related information can indicate data's entry, processing, accuracy, completeness, and integrity.
• This information can be in reports, studies, or interviews with informed people about the data and system; related sources include the GAO or the agency under review.
• A source of information is the organization being reviewed.
• You can often learn initial information about data and data reliability by interviewing agency officials and computer system specialists.
• In addition, agency officials are often aware of evaluations of their computer data or systems and usually can direct you to them.

Making the Data Reliability Determination

• Review one's work periodically and decide whether (1) the data are sufficiently reliable, (2) the data are not reliable, or (3) additional work is needed.
• One is determining data's reliability as needed to support the review's findings, conclusions, or recommendations.
• All factors must be considered, these include:
  • expected importance of data
  • corroborating evidence
  • risk level of using data
  • results of assessment work conducted so far

Including Appropriate Language in the Report

• The report's methodology section should state that analysts have conformed to government auditing standards.
• Standards include the appropriateness of data used.
• Conform to Generally Accepted Government Auditing Standards by discussing data assessment in the report.
• Language depends on whether the data are sufficiently reliable, not sufficiently reliable, or of undetermined reliability.

Appendix 1: Collecting Information for Reliability Assessments

• Document requests and interview questions are provided in order to help evaluate reliability.
• Select or modify items according to your research objectives; focus on specific data elements.
• Review records and data about the system.
• Review it whether it is a large and complicated system or a simple spreadsheet, review it before questioning individuals responsible for and familiar with the data.
• Established systems are likely to have many processes documented.
• Relevant documentation request would include one on a system's purpose and structure.
• Other document request would include information on data elements; relevant also are financial audit reports;

Appendix 2: Sample Interview Questions and Issues Related to Process and System Controls

• An understanding and review of the process and system controls; that is, the structures/programming of the computer system that affects data accuracy/completeness.
• System controls work to reduce the risk of a data mistake.
• Look for those as you formulate your interview question to program and computer system.
• Process examples include the structure, case contorl, training and guidance.
• What are its organizational procedures regarding changes to the system?

Appendix 3: Sample Language for Reporting on Data Reliability

• In a report's intro, paragraphs and section on objectives, scope, and methodology, include a statement about Generally accepted governent auditing statements.
• You conform to GAGAS by speaking in report on how data concerns were evaluated.