[AP]_TT_Interactive_Recommendation_-_Prompt_&_Recommended_Content_Relevancy_Guideline.pdf

[AP] TT Interactive Recommendation - Prompt & Recommended Content Relevancy Guideline​ 1. Background​ TikTok is developing a feature that allows users to adjust video recommendation strategy by proactively providing their interests. There are 2 ways in which they can tell us their interests: survey...

[AP] TT Interactive Recommendation - Prompt & Recommended Content Relevancy Guideline​ 1. Background​ TikTok is developing a feature that allows users to adjust video recommendation strategy by proactively providing their interests. There are 2 ways in which they can tell us their interests: survey and freeform. ​ 1.1 Survey​ Users can identify their interests in certain types of content by clicking options on the survey. Users can identify their interests in certain types of content by clicking options on the survey​ When users can see the survey: The survey will be served immediately after the video if the user performs some selected interactions with the video. Depending on the types of interactions, 2 types of surveys will be shown, i.e. positive or negative. ​ Survey options: There are 4 options on the survey:​ ◦ The 1st option is a fixed one. It always asks users if they'd like to see more/fewer posts that are similar to the current video​ ◦ The 2nd and 3rd options are generated based on the content of the current video. Users can choose to see more of this content.​ ◦ The 4th option is a fixed one that allows users to say whatever they like or dislike​ 1.2 Freeform​ Users can type whatever they wish to see more or less in the freeform editing flow. Once they enter the content, we will send their inputs to AI model to comprehend and transform the input to instructions that our recommendation system can process. We will also show the model output to users so that they confirm if it understands the requests accurately. If yes, they can click [Let's go] to refresh the feed and watch the new content as they wanted. ​ 1.3 Relevant content recall​ After users enter a prompt, we will recall relevant content that matches with the prompt. Same as the above, there are 2 types of prompts that users can use to ask for the content they like: i2i (item-2-item) and entity.​ i2i (item-2-item): More posts like this or Fewer posts like this. In this case, we only focus on More posts like this. After users click "More posts like this" option on the survey, or enter this prompt manually in the freeform editing flow, we will recall content that are alike to the current video that the survey is linked to or where users entered freeform editing flow. ​ Entity: Users can enter in freeform whatever they wish to watch, e.g. Cats, Lady Gaga, etc., and we will recall content that match with user prompt. ​ 1.4 Labeling Objective​ To evaluate the relevancy of the content to user's prompt or to the original post​ 2. Glossary​ Term​ Definition​ User prompt​ The freeform text or input that a user types to express their preferences and requests for the content they wish to see more or less. ​ Post​ The content that users posted in TikTok is called post and its format can be video, photo, live (live streaming) or ads (advertising posts).​ Recalled Content/ ​ The following posts will be pushed to users based on their preference selected in Recommended survey or filled in the freeform.​ Content​ 3. Example​ 4. Discard Rule​ Check the user prompt and discard the case if it meets any one of the following rules. Non-target language: If most words in user prompt do not belong to the target language, please discard the case.​ Unintelligible: Even if most of the words are in the target language, the case should be discarded if the user intention or meaning is hard to discern. ​ Personal data: The user prompt contains personal data in any of the below. Please discard it.​ Personal Data Identifiers​ Notes​ Full name (first and last)​ A single name (or single character in a character based ​ language), such as "bob" or "smith" is not identifiable and does not need to be discarded.​ Public figures:​ 1. Query discussing public figures in the third person do not need to be discarded, as the query is not identifiable to a person (E.g. "Tom Cruise is a good actor."). ​ 2. Query where the author/speaker self-identifies as a public figure should be discarded, as the text/query is identifiable to that person. (E.g.: "This is Tom Cruise speaking.")​ 3. If unsure whether someone is a public figure - Discard.​ Individual phone number​ Does not include phone numbers for businesses. ​ If unsure whether it is for business - Discard.​ Individual email address​ Does not include email addresses for businesses. ​ If unsure whether it is for business - Discard.​ Individual postal address​ Does not include postal addresses for businesses. ​ If unsure whether it is for business - Discard.​ If can identify the full address from a few address identifiers - Discard. Including but not limited to below examples :​  Query with postcode assigned for specific address should be discarded. ​ ID number​ Some ID numbers include both numbers and letters.​ If it only contains the last few words of the ID number, keep the case.​ If unsure whether it is an ID number - Discard.​ Account number for any type of individual/business account (e.g. bank account number, credit card number, loan number)​ Usernames (including public usernames, such as handles for TikTok or other platforms)​ Password/passcode​ 5. Labeling Rule​ Follow the guideline and label i2i and entity prompts separately. ​ 5.1 Type of User Prompt​ 5.1.1 i2i (item-2-item) prompt​ User prompt is categorized as i2i prompt when: ​  Users only ask for "more posts like this" or "fewer posts like this" instead of content with a specific topic like "more travel". ​ It is possible that the user will mention a specific topic like "more posts like this lifeguard". Regard the prompt as i2i if the user intends to recommend similar topics referring to the original videos instead of a new topic. i.e. "More like this (music)" or "more posts like this lifeguard" - i2i​ Type​ User Prompt​ Applied Scenario​ "More", "Love", "Relatable", "Plus"​ I guess it's ok​ i2i (item-2-item)​ More posts like this ​ More posts like this lifeguard​ More posts like this motivational to be the best on my blood line, no distractions​ 5.1.2 Entity Prompt​ User prompt is categorized as entity prompt when 1) Users ask for specific content types, e.g. "More cats", "less news", etc; 2) Besides, there are various aspects that users can specify, including language of content, content format, content creator, etc. ​ The model will firstly understand user's prompts from their sentences and extract the information and match with a list of pre-set aspects and then integrate them into complete instructions. Some of the aspects can be left as "unknown" when a user does not mention them. ​ Type​ Aspects​ Definition​ Possible Model generated Example​ values​ User Input​ User The freeform text or -​ Show me Prompt​ input that a user types to more express their basketball preferences and videos and requests for the content  they wish to see more or motivational less. ​ videos​ Model Open set (Could be anything)​ basketball Output​ Cats​ videos, Entity​ What specific motivational theme/topic​ Yoga​ videos​ blackpink​ etc.​ Closed set​ More​ more​ Action​ To boost or deboost certain types of content​ fewer​ no (when users want "completely no xxx")​ Closed set​ videos​ video​ photo​ Format​ Format of the content​ Live (live streaming)​ Ads (advertisement posts)​ unknown (Default value when the language is not specified)​ Closed set​ unknown​ unknown (Default value when language is not Content The expected language specified)​ Language​ in recommended content​ en (English)​ es (Spanish)​ etc.​ Closed set​ en​ Prompt The used language in en (English)​ Language​ user prompt​ es (Spanish)​ etc.​ Creator​ Content from a certain Closed set​ Unknown​ group of creators​ friends​ follow​  unknown (Default value when format not specified)​ {Action} {Format} {about More Entity} {in Language} {from basketball Creator} in your For You Feed/ and in your For You Page/ FYP​ motivational Combinat integrate entities into videos will be ion​ complete instructions *No sequence required as long as the combined instruction added to your represents all entities with FYP logically sound and coherent throughout.​ 5.2 Labeling Guide​ 1. No. of recommended contents: In most cases, we expect to label 6 recommended posts with the corresponding prompt (or original video), but sometimes there will be fewer (less than 6) recommended videos listed.​ 2. Step 1: Label [Type of Prompt]: To check and label the correct type of user prompt. Label the user prompt as "i2i" or "entity" if it meets the characteristic of two kinds of prompt presented as above. ​ If the type of prompt = "i2i", please label the following tasks: ​ ◦ Step 2: Label [if the original video is clear] ​ ▪ Meaning the original video needs to be perfectly visible and conveys a clear meaning. A blurry image/recording, or a piece of random shoot that shows no clear indication of creator's purpose (text info is not clear either) will have to be excluded from the labeling. ​ ▪ Label as "Yes" if the original video is clear to understand​ ▪ Otherwise label as "No" and procedure ends.​ ◦ Step 2: Label [Recalled Content Relevancy] for each given post​ ▪ The relevancy is based on both text and visual information of the posts. Text info includes description, hashtag, text on videos (text stickers or other formats of embedded text), etc.​ ▪ Label as "Not available" if the video link is not available to open and play.​ ▪ Label as " Relevant" if the text and visual info of the recommended post is relevant to the original post, otherwise label as "Not relevant".  ▪ Label the post as "Uncertain" if the recommended video is not clear in itself or hard to decide if it's relevant to the user prompt. For example, the video is blurry or doesn't have a clear purpose, or the text and visual info in the video are conflicting​ ◦ Step 4: Check [Overall accuracy]​ ▪ This accuracy number represents the proportion of content that is accurately recommended. The formula is "Overall Accuracy = No. of relevant contents / (No. of relevant contents + No. of irrelevant contents)"​ ▪ After the relevancy for all recommended posts has been labeled, the platform will automatically calculate the accuracy rate and categorize it into a range such as 0, 1- 25%, 6-49%, 50%, 51-75%, 76-99% or 100%. Please check if the calculation result is correct and leave comments if the calculation result is wrong.​ ▪ Example: A total of 6 recommended posts for i2i prompt, 3 of them match with the original post, 2 is not relevant and 1 is uncertain. Overall accuracy = 60%(3/5)​ If the type of prompt = "entity", please label the following tasks: ​ ◦ Step 2: Label [If the prompt is clear]​ ▪ Meaning the prompt needs to convey a precise idea without ambiguous, vague or difficult to understand without additional context or clarification. The unclear prompt has to be excluded from the labeling. ​ ▪ Label as "Yes" if the prompt is clear to understand​ ▪ Otherwise label as "No" and procedure ends.​ ▪ Example: More trailer score, label "No" as this prompt does not make sense.​ ◦ Step 2: Label [Recalled Content Relevancy] for each given post​ ▪ The relevancy is based on both text and visual information of the posts. Text info includes description, hashtag, text on videos (text stickers or other formats of embedded text), etc.​ ▪ Label as "Not available" if the video link is not available to open and play.​ ▪ Label as " Relevant" if the text and visual info of the recommended post is relevant with the prompt, otherwise label as "Not relevant". ​ ▪ Label the post as "Uncertain" if the recommended video is not clear in itself or hard to decide if it's relevant to the user prompt. For example, the video is blurry or doesn't have a clear purpose, or the text and visual info in the video are conflicting​ ◦ Step 4: Check [Overall accuracy]​ ▪ This accuracy number represents the proportion of content that is accurately recommended. The formula is "Overall Accuracy = No. of relevant contents / (No. of relevant contents + No. of irrelevant contents)".​  ▪ After the relevancy for all recommended posts has been labeled, the platform will automatically calculate the accuracy rate and categorize it into a range such as 0, 1- 25%, 6-49%, 50%, 51-75%, 76-99% or 100%. Please check if the calculation result is correct and leave comments if the calculation result is wrong.​ ▪ Example: A total of 6 recommended posts for i2i prompt, 3 of them match with the original post, 2 is not relevant and 1 is uncertain. Overall accuracy = 60%(3/5).​ 6. SOP​ 6.1 Interface instruction​ 1. Label/Blind/QA round components​ a. User Prompt​ b. Original Content​ c. Discard? Yes/ No​ d. Type of Prompts: i2i/ entity​ e. If original content or user prompt is clear? Yes/ No/ Not Available​ f. Comment​ g. Overall Accuracy (Automatic calculation)​ h. Recalled content description and post​ i. Recalled content x relevancy: Relevant/ Not relevant/ Uncertain/ Not available​ 2. Screenshot​ 6.2 Operation steps and instructions​ 6.3 Moderate/Blind round​ 1. Start Moderate/Blind and read the user query; ​ 2. Select discard, [Yes] or [No];​ a. If selected discard = Yes , leave your discard reason in the Comment box if any and submit the task;​ b. If selected discard = No, follow the next step;​ 3. Check the type of prompt, [i2i], [entity] or [not clear];​ a. If selected not clear, leave your reason in the Comment box if any and submit the task;​ b. If selected i2i, select if original content is clear, [Yes], [No], [Not available];​ i. If selected No or Not available, leave your reason in the Comment box if any and submit the task;​ ii. If selected Yes, follow the next step; ​ c. If selected entity, follow the next step; ​ 4. Check the recalled content relevancy for each content listed, [Relevant], [Not relevant], [Uncertain], [Not available] according to the guideline;​ 5. The overall accuracy will be automatically calculated and no need to label anything​ 6. Submit the task or submit and leave.​ 6.4 QA round​  Step 1: Start QA;​ Step 2: Copy the task ID in the interface and paste it in the search tool (Ctrl+F) on the alignment sheet to find the corresponding case; ​ Step 3.1: Check if you can find the item ID in the alignment sheet. If yes, follow the next step; If not, please defer the case and move to step 4;​ ◦ Step 3.1.1 :Correct the answers in the QA round, ensuring 100% follow the answers in the alignment sheet;​ Step 4: Submit the task or Submit and Leave.​