Meaning Schema Modeling Guide PDF

Modeling a Meaning Schema is easy with Intent Lab. This section will show the general SLS team approach to Modeling a Meaning Schema. This guide may not be fully applicable for the creation of more complicated Meaning Schemas that require special Modeling or Evaluation. NOTE: For more in-depth m...

Modeling a Meaning Schema is easy with Intent Lab. This section will show the general SLS team approach to Modeling a Meaning Schema. This guide may not be fully applicable for the creation of more complicated Meaning Schemas that require special Modeling or Evaluation. NOTE: For more in-depth modeling information, see go/meaning/concepts Before creating a new Meaning Schema consider using an already existing Meaning Schema from the Meaning Schema catalog. To check the catalog, go to the Query Examples Tool or use Query Debugger to check if your intended queries trigger for some Meaning Schemas. If you want to look at the slots of a Meaning Schema you can look at the Model Tool or the Meaning Explorer Tool If the Meaning Schema does not exists you will need to use Intent Lab(IL). This will require navigating through IL's different tools.and also getting up to date with concepts for new users that may be able to clarify things in the Glossary. Workspace Creation NOTE: You must be in a Workspace to make a new Meaning Schema. Link to workflow on how to create a Workspace. Model Tool (Creating the Meaning Schema) In the Model tool you will create your new Meaning Schema (MS). This means giving it a name, description and defining the slots. IMPORTANT: It is important to have an understanding of how Google defines meaning. Please see go/meaning 1.​ Go to the Model tool. 2.​ Enter the name of your new Meaning Schema (MS) ○​ You can edit this later. 3.​ Click the "Create NameOfSchema" button at the bottom of the dropdown. ○​ You won't see this button if you are not in a Workspace. 4.​ Add a meaningful description to define the scope of the MS ○​ Typically no longer than a sentence. ○​ You can edit this later. 5.​ Add "Search" as the Management Area. The Management area owners will give you approval for submission later. ○​ You can edit this later. 6.​ Click the "Create" button at the bottom of the page. ○​ You must complete the previous 2 steps in order to enable this  7.​ In the left panel enable your MS for Loose Parsing. It is the current way intents are typically triggered in Search. Screenshot 8.​ Whenever you enable a MS for Loose Parser, you should also enable it for offline parsing. Screenshot 9.​ Also in the left panel, do not enable answerless voting unless you are sure that the precision of interpretation generation is sufficiently high to justify this. In some situations, e.g., RiO fulfillment, there are technical reason requiring answerless voting. In these cases, effort should still be taken to ensure that the grammar has sufficiently highly precision of interpretation generation. When answerless voting is enabled, a SxS eval showing more wins than than losses for enabling must be linked in the Workspace Description, and a note tagged answerless_voting must briefly explain the justification (as in this screenshot. Note that in cases touching on Trust and Safety issues, it may be considered a very important win to not produce a potentially harmful answer. In such cases, the net wins vs losses calculation should take into account the imnportance of each win or loss. See go/revised-answerless-voting for rationale. 10.​Add the slots you want to use in your MS. Slots are placeholders so your MS can trigger on queries you do not explicitly define. For example a slot can be for all locations. ○​ You can use KG collections or MIDs. ○​ You can also use other value types for slots that need to be filled with any date or number. ○​ Screencast ○​ IMPORTANT: When adding slots in Meaning Schemas used for Search you must specify if they are required or optional (screenshot) NOTE: Optionally you can use the tabs on the right panel to explore the query examples for your Meaning Schem slots. One tab lets you look at the top traffic patterns constrained by criteria of your choosing. You can add thes your Meaning Schema by marking them as good query examples, however you must remember to import them Once you are done here click the "Expand tool" button on the top right. If you want to review the changes you have made up until this point go to the Workspace Review section of this article. Expand tool (Adding Patterns) This is the tool users typically use to add all patterns that should match your Meaning Schema in a Search stack. It is important to have defined the slots for your Meaning Schema by the time you get to the Expand tool. Any patterns you add here will receive the "induction" data tag, meaning that they will be used in inducing and training the Grammar that will trigger on a Search stack.  1.​ Go to the Expand tool and navigate to your Meaning Schema. Make sure the filters are set to the correct Workspace. 2.​ Optional: If you added query examples in the Model tool be sure to import them using the prompt on the top of the page. 3.​ Add a new pattern by clicking on “+”. ○​ You will have to provide an example for what the slot could resolve to. Screenshot ○​ Separate patterns by line in order to add multiple patterns at once. A location slot could resolve to "New York". 4.​ Add a couple more initial patterns. 5.​ Mark the patterns as good. Marking a pattern as good means it will trigger your Meaning Schema on a search stack. Marking a pattern as bad will blocklist the pattern. 6.​ Use the "Fetch and Rank" functionality. This will suggest patterns based on what you have already marked as good. ○​ Predicted patterns will be light green. Currently prediction is based on Loose Parsing. If something is predicted in the Expand tool it will also trigger your Meaning Schema in production. Therefore, you do not need to explicitly mark predicted patterns as good. ○​ You can continue to "Fetch and Rank" as many times as you wish. It will continue to use the information from queries you have thumbed-up or thumbed-down to find related queries. NOTE: Patterns you mark up or down in the Expand tool will receive the induction tag, meaning it will trigger you a search stack. Once you have successfully added all the patterns within the scope of your Meaning Schema you will have to generate KScorer salient terms for them. If you want to review the changes you have made up until this point, go to the Workspace Review section of this page. KScorer salient Terms If your Meaning Schema is new you will need to generate and submit salient terms for the related Meaning Schema before you can submit your Meaning Schema Workspace. Salient terms contribute to the KScorer Model, the secret sauce behind Googles ranking algorithm. Please read this section for general knowledege on KScorer in IL. WARNING: If you make changes to the patterns that trigger for your Meaning Schema you will have to regenerat Evaluations  You need to run evaluations on your changes to make sure you don't break Search. Note: Read about KE Evals and SxS at: Overview: Evals and KE Quality. This is a great resource to learn how any evaluations work. Users can run evals from the Job Runner tool. You will run the following evals: 1.​ KE SxS or KE SxS with fallback fulfilment ○​ This eval shows side by side differences between the production version of Search against the version of Search with your Meaning Schema changes in it. ○​ This eval lets us see the potential impact on traffic. ○​ If you don't have fulfillment set up, use fallback fulfillment. A fake fulfillment in the form of a OneBox will be displayed in your SxS. Having fulfillment increases the amount a query triggers. This will help see if your Meaning Schema will overtrigger once it has fulfillment. ○​ You should include your KScorer salient term CL in this eval. ○​ This eval generates SxS diffs that can be rated via Furball. 2.​ KE Eval ○​ This is a regression test to make sure you are not going to break anything when submitting the Meaning Schema. ○​ You should include the KScorer salient term CL in this eval. ○​ There will not be fake fulfillment on this Eval. 3.​ Load test ○​ Load test is run to make sure you are not increasing the load to unacceptable levels during the annotation stage of a query. ○​ You can kick this of from the Intent Lab Job Runner tool ○​ Link to Load Test documentation Workspace review Navigate to the Workspace Review tool. You will be able to see all your changes in the respective tabs. Note that patterns you added in the Expand tool are now shown as query examples with annotated slots on them, and they also have the "induction" data tag. Description To change the Workspace description press "Modify" on the top right. When creating a new MS you need to add the links to the evals so the reviewer can easily see the impact of your changes. The Workspace description is split up into 2 parts the description and the tags. The tags will be considered later when you run presubmit tests during submission. Here is a typical format. Description: Unset Workspace Canonical Name (grammar creation for 3 MS) MS Creation for: MS1, MS2, MS2 Tags: Unset DIFFS_APPROVED= IMPACT_AT_100= KE_EVAL= QREWRITE_LOAD_TEST=` NOTE: Tags are automatically updated when an eval completes. Send for review You will need to send the Workspace to be approved by the person who owns the Management Area for the MS you are working on. If you are following this flow it will most likely be the "Search" Management area. 1.​ Click request review on the top left 2.​ Select the approver from the menu and click send for review. Screenshot ○​ This will automatically assign a reviewer. TIP: If you want the Minecraft team to look at your Workspace before the final reviewers, send an email to [email protected] with a link to the workspace. The reviewer will leave comments on the Workspace. You should address these appropriately. Sometimes the reviewer will ask you to remove certain induction examples or add some missing ones. You may possibly have to remove a collection in a slot or even add a new slot. WARNING: If you make any changes to the Meaning Schema or the related "induction" examples you will need to evaluations. Submission Once your Workspace has been approved you may submit it. Submitting takes a moment because there are a couple of presubmit checks that need to take place. If the pre-submit checks cause the submission to fail, the failures will need to be addressed. TIP: One common issue is invisible diffs. See this page if you have that issue. When will grammar reach production? Various binaries need to release new versions in order for you changes to be available in production. Use: ​ go/pp-where-is-my-cl ​ go/pp-where-is-my-cl-dashboard to help you track the release of your changes to production.