Podcast
Questions and Answers
What is the primary objective of the 8D evaluations?
What is the primary objective of the 8D evaluations?
Which of the following is NOT a criterion for rating responses in the 8D evaluations?
Which of the following is NOT a criterion for rating responses in the 8D evaluations?
When evaluating a response, what should be particularly checked in relation to external sources?
When evaluating a response, what should be particularly checked in relation to external sources?
What is the significance of the hierarchy of cooperation types in the 8D evaluations?
What is the significance of the hierarchy of cooperation types in the 8D evaluations?
Signup and view all the answers
What does a safety refusal indicate in the context of 8D evaluations?
What does a safety refusal indicate in the context of 8D evaluations?
Signup and view all the answers
Which task type can evaluators expect to encounter in the 8D project?
Which task type can evaluators expect to encounter in the 8D project?
Signup and view all the answers
What should evaluators do if they find suspicious sources listed in the references?
What should evaluators do if they find suspicious sources listed in the references?
Signup and view all the answers
In what context are the Facebook feed tasks rated?
In what context are the Facebook feed tasks rated?
Signup and view all the answers
What is the primary focus when evaluating the latest response in a conversation history?
What is the primary focus when evaluating the latest response in a conversation history?
Signup and view all the answers
What should be done if the task involves an image that is necessary for evaluation but does not display?
What should be done if the task involves an image that is necessary for evaluation but does not display?
Signup and view all the answers
In tasks where the model cites sources, what must be verified?
In tasks where the model cites sources, what must be verified?
Signup and view all the answers
Which of the following describes Facebook Feed Context tasks?
Which of the following describes Facebook Feed Context tasks?
Signup and view all the answers
What should be considered when using the SRT platform?
What should be considered when using the SRT platform?
Signup and view all the answers
Which instruction is NOT part of the task workflow on the SRT platform?
Which instruction is NOT part of the task workflow on the SRT platform?
Signup and view all the answers
What should be prioritized when reviewing chat history and citations?
What should be prioritized when reviewing chat history and citations?
Signup and view all the answers
When dealing with a safety refusal, what should be ensured in the response?
When dealing with a safety refusal, what should be ensured in the response?
Signup and view all the answers
Study Notes
8D Project Overview
- Evaluate AI chatbot responses to user prompts
- Rate the prompt based on 8 criteria:
- Accuracy
- Citation Correctness
- Instruction Following
- Refusal
- Grammar/Presentation
- Relevance
- Tone/Style
- Comprehensiveness
- Focus on whether the response addresses linked sources or social media posts in the conversation history
- Tasks may not have refernce links or social media posts, but check each chat history for them
- Higher quality responses effectively address and/or incorporate information from external sources of information
- Quality is of the utmost importance
Important Documents and Resources
- SRT (Customer Platform) Interface Preview: Familiarize yourself with the different parts.
- Key SRT Terms
- Remotasks/Outlier Interface Preview: Familiarizes yourself with where to input information into the task interface.
- Key Remotasks/Outlier Terms
- SRT Login Instructions
- Task Workflow: You will evaluate response to a prompt and be given conversation history as context on Remotasks/Outlier and SRT Platform.
Benchmark Exam
- Complete benchmark training tasks
How to Complete Specific Tasks
-
Facebook Feed Context: These tasks are fairly common where the latest model response must be relevant to the most recent Facebook Feed Context.
- Example: Facebook Feed Context tasks show up fairly often on the project.
Example Task of Prompt and Response with Citations
- Example task of Prompt and Response with Citations:
- The model can cite documents.
- Example document ID: []
Response Grading Rubric
- Tone & Style: Refresher Course available.
-
Refusals:
- Safety Refusal: Explicit mention of something along the lines of "I'm not supposed to answer that".
- Capability Refusal: "I don't understand this prompt."
- Ignorance: "I don't know that"
- Other Refusal: This is a refusal that does not fit into the other defined categories.
Appendix
Good Justification
- Example of acceptable review notes:
How to Rate Facebook Feed Tasks (One-Feed Queue)
-
The Basics:
- Example with rating and explanation/thought process:
- Chat History and latest user prompt:
- Facebook Graphic:
- References:
- Response:
- Rating with Explanations:
Instructions Update Log
- Important Project Updates:
- Use Discourse Blockers Google Form to request help regarding the Discourse platform
Important Project Updates
- If any source looks suspicious, follow up with independent fact checking
Studying That Suits You
Use AI to generate personalized quizzes and flashcards to suit your learning preferences.
Description
This quiz focuses on the evaluation of AI chatbot responses based on eight essential criteria such as accuracy, relevance, and tone/style. Participants will learn how to assess the effectiveness of chatbot communication and incorporate information from external sources. Quality evaluation is emphasized throughout the process.
