Maps Search Evaluation Guidelines PDF

Maps Search Evaluation Guidelines September 2023 1. Introduc on........................................................................................................

Maps Search Evaluation Guidelines September 2023 1. Introduc on..............................................................................................................................................5 1.1.The Rating Tool....................................................................................................................................................................................................5 1.2.Rating Work low...............................................................................................................................................................................................11 1.3.Rating Interface................................................................................................................................................................................................13 1.3.1.Query-Level Navigational Result Question.................................................................................................................................................14 1.3.2.Result-Level Rating Checkboxes...................................................................................................................................................................14 1.3.3.Result Relevance Rating..................................................................................................................................................................................15 1.3.4.Result Relevance Demotion Checkboxes...................................................................................................................................................17 1.3.5.Data Accuracy...................................................................................................................................................................................................17 1.3.6.Comments..........................................................................................................................................................................................................17 2. Understanding User Intent.......................................................................................................................18 2.1.Query Types.......................................................................................................................................................................................................18 2.1.1.Research Expectation.......................................................................................................................................................................................22 2.2.Result Types.......................................................................................................................................................................................................22 2.3.Location Intent..................................................................................................................................................................................................23 2.3.1.Explicit Location................................................................................................................................................................................................24 2.3.2.Implicit Location..............................................................................................................................................................................................24 3. Ra ng the Query-Level Naviga onal Result Ques on...............................................................................25 4. Ra ng Result-Level Issues........................................................................................................................25 4.1.Result Name/Title in Unexpected Language or Script.....................................................................................................................25 4.1.1.Name of Business/POI Results........................................................................................................................................................................27 4.1.2.Title of Address Results....................................................................................................................................................................................30 4.2.Business/POI is Closed or Does not Exist.............................................................................................................................................30 4.2.1.Closed/Does Not Exist vs. Inaccurate Name and Address......................................................................................................................31 4.2.2.Rating Relevance of Closed/Non-Existing Business/POI.......................................................................................................................31 4.3.Business/POI Status is PERMANENT_CLOSURE................................................................................................................................31 4.3.1.No Status Shown..............................................................................................................................................................................................32 4.3.2.Status Shown as PERMANENT_CLOSURE.................................................................................................................................................32 5. Relevance................................................................................................................................................32 5.1.Query-Result Connection..............................................................................................................................................................................33 5.1.1. General Connection.........................................................................................................................................................................................34 5.1.2. Abbreviation/Alternate Name Connection..............................................................................................................................................34 5.1.3. Category Connection.....................................................................................................................................................................................34 5.1.4. Spell Correction Connection.........................................................................................................................................................................35 5.1.5. Transit Intent Connection..............................................................................................................................................................................35 5.1.6. Special Character Connection......................................................................................................................................................................35 5.1.7. Address-Result Connection............................................................................................................................................................................36 5.1.8. Lack of Connection.........................................................................................................................................................................................36 5.2. Satisfying User Intent....................................................................................................................................................................................38 5.3.Prominence.........................................................................................................................................................................................................39 5.4.Distance................................................................................................................................................................................................................40 5.5.Distance vs. Prominence...............................................................................................................................................................................42 5.6.Many Possible Results....................................................................................................................................................................................42 5.7.Few Possible Results.......................................................................................................................................................................................45 5.8.Few Possible Results and Greater Distance..........................................................................................................................................46 5.9.Rural Areas.........................................................................................................................................................................................................47 5.10.Location – User Intent Deviation............................................................................................................................................................48 5.11.Rating for User Inside and Outside Viewport...................................................................................................................................49 5.12.Partial Address Does Not Exist................................................................................................................................................................51 5.13.City vs. Municipality.....................................................................................................................................................................................52 5.14.Unexpected Results......................................................................................................................................................................................52 5.15. Multiple Transit POIs with the Same Name......................................................................................................................................54 ti ti ti f ti ti 5.16.POIs and Transit Intent...............................................................................................................................................................................55 5.16.1.Transit Queries.................................................................................................................................................................................................56 5.16.2.Stops vs. Stations............................................................................................................................................................................................59 5.17.Parking Intent.................................................................................................................................................................................................61 5.18.Relevance: Service-Level Mismatch......................................................................................................................................................61 5.19.Rating Relevance When Result Status is PERMANENT_CLOSURE...........................................................................................64 5.19.1.Expected PERMANENT_CLOSURE Status..................................................................................................................................................64 5.19.1.1.Expected PERMANENT_CLOSURE Status: Navigational Query......................................................................................................66 5.19.1.2.Expected PERMANENT_CLOSURE Status: All chains closed in area............................................................................................68 5.19.2.Unexpected PERMANENT_CLOSURE Status............................................................................................................................................69 5.19.2.1.Unexpected PERMANENT_CLOSURE Status: Open Options are Nearby....................................................................................70 6. Name Accuracy........................................................................................................................................71 6.1.Name Not Applicable (n/a).........................................................................................................................................................................71 6.2.Result Names.....................................................................................................................................................................................................72 6.2.1. Correct Name....................................................................................................................................................................................................72 6.2.2. Partially Correct Name..................................................................................................................................................................................75 6.2.2.1. Minor and Moderate Misspellings and Missing or Extra Words.....................................................................................................75 6.2.2.2. Service-Level Mismatch....................................................................................................................................................................................78 6.2.2.3. Holding Name and Corporate Structure....................................................................................................................................................79 6.2.3. Incorrect Name................................................................................................................................................................................................80 6.3 Result Classi ication........................................................................................................................................................................................82 6.3.1. Correct Classi cation......................................................................................................................................................................................82 6.3.2. Incorrect Classi cation..................................................................................................................................................................................83 6.4. Can’t Verify.........................................................................................................................................................................................................85 6.5 Transit Names...................................................................................................................................................................................................86 6.6. Parking Names.................................................................................................................................................................................................86 6.7. Final Name Accuracy Rating (Summary Table).................................................................................................................................88 7. Address Accuracy: Components and Issues..............................................................................................89 7.1.Address Components.....................................................................................................................................................................................89 7.1.1.Street Number.....................................................................................................................................................................................................89 7.1.2.Unit/Apt................................................................................................................................................................................................................91 7.1.3.Street Name........................................................................................................................................................................................................92 7.1.4.Sub-Locality and Administrative Subdivisions..........................................................................................................................................93 7.1.5.Locality.................................................................................................................................................................................................................94 7.1.5.Region or State...................................................................................................................................................................................................94 7.1.6.Postal Code.........................................................................................................................................................................................................95 7.1.7.Country.................................................................................................................................................................................................................95 7.2.Address Does Not Exist.................................................................................................................................................................................96 7.3.Language/Script Issue in Address............................................................................................................................................................96 7.4.Country-Speci ic Issue...................................................................................................................................................................................97 7.4.1.Country-Speci c Issue vs. Correct with Formatting Issue.......................................................................................................................97 7.5.Other Issue..........................................................................................................................................................................................................97 7.6.Correct with Formatting Issue...................................................................................................................................................................98 7.7.Can’t Verify..........................................................................................................................................................................................................98 7.7.1.Can’t Verify for Street Number........................................................................................................................................................................99 7.7.2.Can’t Verify for Street Name...........................................................................................................................................................................99 8. Address Accuracy: Result Expecta ons...................................................................................................100 8.1.Business/POI Result....................................................................................................................................................................................100 8.1.1.Business/POI Address Research and Resources.......................................................................................................................................100 8.1.2.Alternative O cial Addresses......................................................................................................................................................................101 8.1.2.1.Department Addresses....................................................................................................................................................................................101 8.1.2.2.Result Missing the Street Address..............................................................................................................................................................103 8.1.2.3.P.O. Box, Mailing Addresses, and Management Of ices......................................................................................................................104 8.1.3.Businesses/POI with Moving Locations....................................................................................................................................................105 8.1.4.Events.................................................................................................................................................................................................................105 8.2.Address Type Results..................................................................................................................................................................................106 2 ffi fi f f fi fi ti f  8.2.1.Address Result..................................................................................................................................................................................................106 8.2.2.Address Formats............................................................................................................................................................................................106 8.2.3.Street Result....................................................................................................................................................................................................107 8.2.4.Locality Result.................................................................................................................................................................................................107 8.2.5.Postal Code Result.........................................................................................................................................................................................108 8.2.6.State/Region/Territory Result.....................................................................................................................................................................108 8.2.7.Country Result.................................................................................................................................................................................................108 8.3.Features Without an Expected Address..............................................................................................................................................108 8.3.1.Natural Features.............................................................................................................................................................................................108 8.3.2.POIs Without an Expected Address...........................................................................................................................................................110 8.3.2.1.Minimum Address Component....................................................................................................................................................................110 8.3.2.2.Of icial Address Present..................................................................................................................................................................................111 8.3.2.3.No Of icial Address............................................................................................................................................................................................113 9. Pin Accuracy..........................................................................................................................................115 9.1.General Considerations..............................................................................................................................................................................116 9.1.1.Pin Resources.....................................................................................................................................................................................................116 9.1.2.Satellite Imagery vs. Vector Map.................................................................................................................................................................116 9.1.3.Boundaries of the Feature.............................................................................................................................................................................117 9.1.3.1.Half ’n Half and Tennis Rules.........................................................................................................................................................................118 9.1.3.2.Boundaries in Shared Spaces........................................................................................................................................................................120 9.1.4.Next Door..........................................................................................................................................................................................................122 9.1.5.Can’t Verify (Smallest Identi ed Area).......................................................................................................................................................123 9.2.Single Rooftop.................................................................................................................................................................................................124 9.2.1.Leaning Buildings...........................................................................................................................................................................................139 9.2.2.Residential Property with Multiple Buildings.........................................................................................................................................141 9.3.Multiple Rooftops..........................................................................................................................................................................................142 9.3.1.Multiple-Rooftop Rating...............................................................................................................................................................................143 9.3.2.Campus Rating...............................................................................................................................................................................................149 9.3.2.1.Single-Complex Campus..................................................................................................................................................................................151 9.3.2.2.Dispersed Campus.............................................................................................................................................................................................154 9.4.Features Without a Rooftop......................................................................................................................................................................157 9.4.1.Natural Features..............................................................................................................................................................................................157 9.4.2.Administrative Divisions..............................................................................................................................................................................160 9.4.3.Streets................................................................................................................................................................................................................161 9.5.Parking Lots and Structures.....................................................................................................................................................................163 9.6.Transit POIs.....................................................................................................................................................................................................164 9.6.1.Bus, Tram, and Streetcar Stops....................................................................................................................................................................166 9.6.2.Transit POI with Multiple Rooftops/Platforms.......................................................................................................................................167 9.6.3.Transit POIs with a Single Rooftop............................................................................................................................................................170 9.6.4.Underground Transit Stations....................................................................................................................................................................170 9.6.4.1.How to Establish an Entrance Polygon.....................................................................................................................................................171 9.6.4.2.Examples................................................................................................................................................................................................................172 10.How to Rate Results..............................................................................................................................176 10.1.Speci ic Address..........................................................................................................................................................................................176 10.2.Non-Speci ic Address................................................................................................................................................................................191 10.3.Query Address Does Not Exist..............................................................................................................................................................199 10.4.Point of Interest...........................................................................................................................................................................................201 10.5.Understanding Multiple Query Interpretations: Beyond Viewport and User Location...............................................205 10.6.Business Queries.........................................................................................................................................................................................208 10.6.1.Non-Chain Businesses.................................................................................................................................................................................209 10.6.2.Chain Businesses..........................................................................................................................................................................................212 10.6.3.Chain Business with Location Modi er..................................................................................................................................................216 10.6.3.1.General Location Modi ier...........................................................................................................................................................................216 10.6.3.2.Speci ic Location Modi ier...........................................................................................................................................................................224 10.6.4.Back O ces and Businesses with no Physical Location...................................................................................................................228 10.6.5.Business/POI does not Exist......................................................................................................................................................................230 10.7.Category Queries........................................................................................................................................................................................231 3 f f f f ffi f f f fi fi  10.7.1.Demotion for Distance with User Inside Viewport...............................................................................................................................232 10.7.2.Demotion for Distance with User Outside Viewport..........................................................................................................................234 10.7.3.Demotion for Distance with Fewer Possible Results...........................................................................................................................236 10.7.4.Category Queries and User Intent...........................................................................................................................................................238 10.7.5.Category Queries and Prominence.........................................................................................................................................................239 10.7.6.Category Query with Location Modi er................................................................................................................................................239 10.7.7.Navigational Results for Category Queries............................................................................................................................................241 10.7.8.Clear Categories...........................................................................................................................................................................................245 10.7.9.Soft Categories..............................................................................................................................................................................................246 10.7.10.Category Query: Parking...........................................................................................................................................................................251 10.8.Rating Results with PERMANENT_CLOSURE Status...................................................................................................................255 10.8.1.Expected PERMANENT_CLOSURE Status...............................................................................................................................................255 10.8.2.Unexpected PERMANENT_CLOSURE Status........................................................................................................................................256 10.9.Other Query Types.....................................................................................................................................................................................258 10.9.1.Routing Queries............................................................................................................................................................................................258 10.9.2.Coordinate and “My Location” Queries..................................................................................................................................................260 10.10.Unclear Results.........................................................................................................................................................................................262 11.Top Ra ng Tips......................................................................................................................................269 11.1.What’s the relevance when query is business/POI name and address and the result is only the address?.......269 11.2.What’s the relevance when the query is a street name and the result is a single business on that street?........269 11.3.How do I know what the location intent is when user is outside/inside a fresh viewport?......................................270 11.4.How do I know what the location intent is when user is outside/inside a stale viewport?......................................271 11.5.This full address result does not exist. How do I rate it?...........................................................................................................271 Appendix: Release Survey..........................................................................................................................273 4 ti fi 1. Introduction Welcome to Maps Search Evaluation. These guidelines will teach you how to use the evaluation tool and rate Maps Search results. For each task, you will be presented with a search query and the results that are provided to Maps users. The goals of this task are: Search Relevance and Search 2.0: To determine how well the results satisfy the user’s query (result relevance) Search 2.0 only: To check the data accuracy (name, address, and pin location) of the results 1.1. The Rating Tool This is the web-based rating tool used to rate Search 2.0 and Search Relevance queries. 5 Anatomy of the Tool Name Screenshot Description The task bar shows: Task Type: The name of the task. Task ID: A number that identifies the specific task item. Use this number whenever you communicate about a task or rating. Task Bar Request ID: A reference number for internal use only. Do not use this number to communicate about a task or rating. Estimated Rating Time: Time to rate all results Rating The Rating Guidelines shortcut button Guidelines opens the guidelines for this task. Allows you to skip a survey in rare cases Release where a technical or other issue prevents Survey you from rating. See Appendix for complete instructions. Viewport shortcut buttons provide convenient controls to pan or zoom to common map viewport settings. Show Show User User moves the result viewport to the user’s location without changing the zoom level (map scale). Show Viewport moves the result Show viewport to the center of the user Viewport viewport and zooms in on the viewport area. Use Show All to return from any zoom Show All level to the initial display showing the user, the viewport, and all the results. 6  Name Screenshot Description The query header shows: Query: The term the user entered into the search bar. Viewport Age: How long ago the user moved the viewport. A fresh viewport is indicated by the word FRESH displayed in green text. Query A stale viewport is indicated by the Header word STALE displayed in red text. Locale: The country and language combination this query is for. Country: Should match the country in the Locale field. If it does not, defer to the country found in the Locale field. User Lat,Lng: The user’s location. Top-Level A top-level question that must be Question answered once for each query. The result returned for a user query. The color of the result heading corresponds to the color of the pin on the map that marks that result’s location. (See Result Pin on page 9.) The result address, classification and status (if included) will be shown beneath Result the result name/title. (The classification of the result refers to its category.) You will evaluate the relevance of each individual result. If you are rating Search 2.0, you will also rate the name/address/ pin accuracy for each result. Note: A query can generate more than one result. 7  Name Screenshot Description A result may also display its status. If the status appears and is set to Status PERMANENT_CLOSURE, the result requires special treatment when rating relevance. ` You may occasionally see one or more normal looking or grayed-out results with Result with no rating fields beneath them. No rating no Rating is required for these results. This is Choices expected and is not a technical issue or a reason to release the survey. User viewport: A transparent purple box centered over a device icon shows the user’s viewport at the time the query was issued (if known). The user viewport represents the map area the user was looking at when making the query. The shape of the viewport may vary depending on the size and orientation of User the device being used. Viewport Note: In some cases, the user viewport may be so small that it cannot be seen, but (if the viewport information is available) the device icon will always be visible, regardless of the map scale. Note: The viewports you see in the examples in these guidelines may be grey. 8  Name Screenshot Description This device icon shows the center of the user viewport at the time of the user query. The purpose of the device icon is to make it easier to find the user viewport. It does not indicate the user's location. This icon should not be used for Device Icon measuring distance. To measure distance, start from the outer edge of the user viewport. This icon has no rating significance. Its only purpose is to help you find the viewport on the map. A blue and white icon indicates the user location at the time of the query (if known). User Location Note: Clicking on the icon reveals the coordinates (latitude/longitude) of the user’s location. A pin indicates a result’s location. The head of the pin is only an indicator. The tip marks the actual location. Each pin on the map has a unique number and color, which matches the Result Pin number and color of its corresponding result heading.(See Result, above) Note: The pins you see in the examples in these guidelines may look different from the pin shown in this chart and the ones in the tool. 9  Name Screenshot Description Clicking any individual result or pin causes a pop-up box to appear over the pin. The text within the box may show additional information and should only be used for research purposes to help understand the result: Pop-up Information Phone number (for research only, Box never call a business) URL Coordinates (latitude/longitude) Note: Do not rate the information in the pop-up box. Rate only the information in the result. Map Layer Allows you to select which map layer is Selector displayed. Allows zooming in and out on the map. Zoom Change the zoom level by clicking the + Controller (zoom in) or – (zoom out) buttons. Copy to Click to copy information to a clipboard Clipboard Click the pin icon at the top right of the map to place a pin anywhere on the map. Drop Location Pin A purple pin and a pop-up window showing the pin’s address and latitude/ longitude will appear. 10  Name Screenshot Description To measure the distance, click the ruler icon at the top right of the map. Measure Distance You can now drop a start pin (green) and an end pin (red). The tool will display the distance between them. Draw Tool Click to draw custom polygons. Click Submit to save your ratings and Submit display the next available task. 1.2.Rating Workflow Complete each step of the following workflow before you assign a rating. Evaluate and rate each rating drop-down independently of the others. Rate result Relevance based on whether the result fits the query intent, even if the result is closed or does not exist or has incorrect Name, Address, or Pin data. If you are rating Search Relevance you will stop after you’ve rated result relevance. If you are rating Search 2.0, you will continue by rating Name Accuracy, Address Accuracy, and Pin Accuracy individually. Note: For any result in a Search 2.0 project, you may be asked to make all of the possible ratings, some of them, just one, or none. This is expected and not a reason to escalate or release a task. Rate each result on its own merits. Do not demote duplicate or very similar results. 11 12 1.3. Rating Interface Your ratings and comments will be collected in the rating interface pictured below. You will first answer one top-level question about the query itself, then rate each individual result using the other checkboxes and drop-downs. 13 1.3.1. Query-Level Navigational Result Question You will answer a top-level question for each query to indicate whether there are any results in the real world that would completely satisfy the user intent. Query-level question Explanation Notes Is there a navigational result Are there any results in the real You will only need to answer for this query? world that can completely this question once for each satisfy the user intent? query. Answer to Query-Level Query Explanation Question [Eiffel Tower] Yes There is only one result in the real world that can completely satisfy the user intent. [Starbucks] No There is not only one result in the real world that can satisfy the intent of this query. Note: If you are presented with a query that has no results, you must still answer this question and submit your answer. 1.3.2. Result-Level Rating Checkboxes For each result, you will be presented with two high-level questions. These will allow you to highlight any problems with the result. High-level Explanation Notes question Result is not in the language of the If you check this box, your rating for this Result name/ query, the test locale, or the result result will be complete (all other questions title is in region (the expected languages) or is will be unavailable). unexpected in a mix of expected and unexpected language or languages (unless it is an official script company, chain, or brand name). 14  High-level Explanation Notes question If you check this box, only the relevance needs to be rated. Pay attention to the status of the POI. If the result’s status is PERMANENT_CLOSURE, refer to 5.19 Rating Relevance when Status is set to PERMANENT_CLOSURE and 10.8 Permanently Closed Results for rating Business/POI Research shows that the business is instructions. is closed or closed or that no such business exists does not exist at the address listed in the result. If the status is blank or displays any other text, rate the relevance of the result as if it did exist. Use thorough research and your best judgement to determine whether the result is closed, has never existed, or the result component(s) are incorrect.See Unclear Results for further guidance. 1.3.3. Result Relevance Rating The relevance rating will take into consideration user intent as well as distance/prominence. Relevance is always rated independently of any data (name/classification, address, or pin) inaccuracies. This means that when rating relevance we always assume that the result exists (even if research reveals the location is closed) and that the data presented is correct. Always rate against the real world: If there is a better result available but it is not shown, demote the existing result(s) while considering the missing one(s). 15  Relevance Explanation Notes Rating The most likely result implied by the user’s query, location, and/or viewport that completely satisfies the user’s distinct intent. A distinct user intent is defined by a A rating of Navigational for one combination of the following result does not prevent other Navigational characteristics: results from satisfying the query to Extreme prominence (e.g. Eiffel Tower) a lesser degree. Uniqueness (e.g. complete address) Proximity (e.g. incomplete address that points to an unambiguous option that is is extremely close) Highest possible initial rating for A high-quality result that clearly satisfies the ambiguous queries as well as for Excellent user’s intent. Multiple results can be queries that are not eligible for a considered high quality. Navigational rating. Only partially satisfies the user’s intent due Good to relevance, prominence, or distance. Any relevance rating of Good or Technically satisfies the user’s intent but below requires an indication of the Acceptable does so poorly due to relevance or distance. reason(s) for the demotion: User Intent Issue Does not satisfy the user’s intent due to lack Distance/Prominence Issue Bad of relevance or great distance (when closer satisfying results are available). 16 1.3.4. Result Relevance Demotion Checkboxes If you give a result a Relevance rating of Good or below, you will be presented with two checkboxes. Use them to select the reason(s) for your rating: User Intent issue Distance/Prominence issue You must also leave a comment explaining your rating. Checkbox Explanation Notes User Intent Select when the demotion is due to a result only issue partially fulfilling the query intent. Select when the demotion is due to a distance issue based on the user or viewport location or If both issues were reasons for Distance/ when you demoted for a result that is less your relevance rating, select both Prominence prominent based on the initial query intent. You checkboxes. issue can also select this option when the query asks for something at a specific location and the result is not at or near the requested location. 1.3.5. Data Accuracy If you are rating Search 2.0, once you've rated relevance, you’ll evaluate three result components for data accuracy: Name, Address, and Pin. 1.3.6. Comments Comments are mandatory for any result relevance rating of Good or below and for any data issues not rated Correct/Perfect and should explain the reasons for all rating demotions. Be sure to: Include user intent: Indicate what you think the user intent is. Refer to the guidelines: If the demotion is due to a guideline instruction, note the guideline section you used to make your decision. Include sources: If you find incorrect information in a result, your comment should include the correct information and its source. Be sure that any links you include lead directly to the information you provide. If they don’t, include a link to the source you used and explain how to get to the information from there. Use URL shorteners when inserting long links into your comments Be concise: Don’t write elaborate comments. Be short and specific. Comment in English: Your comments will sometimes be reviewed or used by people who are not familiar with your test locale or its language. Comments in English can be understood by the widest possible audience. 17  Issue Expectations Example Rating and Comment The comment should explain the Relevance -> Good (Distance/ reasoning behind your rating. A demotion Prominence) Relevance may be due to distance from the user’s Comment: Demoted -1 for a few closer Demotions location compared to similar results/ real world locations in the Middletown/ potential results, secondary intent, or Anchorage area other reasons. Address Accuracy -> Incorrect — Street Number Include a comment when you rate a Comment: Correct street number is: 332 Name, Address, or Pin something other W El Camino Real, Sunnyvale, CA 94087 than Correct or Perfect or you give a Data Issues as found on the official site: https:// rating of Closed/Does not Exist or www.patelco.org/locations/sunnyvale Unexpected Language. Pin Accuracy -> Wrong Comment: Wrong pin: correct location is: 52.368067, 4.895389 2. Understanding User Intent Before evaluating any results, you’ll need to understand what the user wanted to find when they typed in their query. To determine user intent, use a combination of your own research and local knowledge plus the information found in the rating interface, which includes: User location User viewport Viewport age User intent refers to the interaction between query, user location, user viewport, and viewport age. 2.1. Query Types The queries you are evaluating are real queries submitted by users. Because they are real, they can: Be misspelled Contain mistakes Be inconsistent Be ambiguous Be badly formatted Be, or contain, emojis Have no maps intent The first step in the rating process is to make sure you understand the query. If you see a foreign-language query, do not release the survey. Research the query or use an online translation tool to find out what it means, then rate it as usual. 18 You will see several types of queries, including: Address queries Address queries contain all or part of a complete address, including street number, street name, locality, state, country, and postal code. Query Explanation [717 E El Camino This is an address query in the form of a full address. It contains a street Real, Sunnyvale, number, street name, locality, state and postal code. CA 94087] [Stevens Creek This is an address query for a street. The user is looking for the location of Blvd, Cupertino this particular street. CA] This is an address query for a country. The user is looking for the location of [Ireland] the country of Ireland. This ambiguous query could refer to New York city or New York state. You [New York] will find more information on prominence, user location, and the viewport in the following sections to help you determine the intent of this query. Point of Interest (POI) queries A point of interest (POI) is a location that people find interesting or useful. POIs can be locally or internationally known. These guidelines use a very broad definition of POI in which businesses are also considered POIs. Query Explanation [London Bridge] This is a query for a famous POI in London. [Danube River] This is a query for a river in Europe. [Charing Cross This is a query for a transit station in London. Station] [Union Square, SF] This is a query for a location in San Francisco. 19 Business queries Business queries contain the name of a specific business. Query Explanation This is a business query with a location modifier (Palo Alto). The user is [Zola Palo alto] looking for a specific restaurant (Zola) in Palo Alto, CA. Queries can include location modifiers, like city names, that make them more specific. This is a business query with a request for a specific store in Sunnyvale, CA. [Bookasaurus] The viewport and user location can help pinpoint the desired location for such a query. [Starbucks, 7 This is a query for a chain business followed by a specific address. Chain Boulevard businesses are businesses that have more than one location, and include Poissonnière, everything from national chains, like Starbucks, down to small local chains, 75002 Paris, like Burrito Boyz. Target, Auchan, Albert Heijn, and Boots are all chain France] businesses. This is a business query with a request for the chain business Target and [Target sunnyvale] location modifier of Sunnyvale, CA. Category queries A category refers to a group of entities that share a set of characteristics. Query Explanation This query is for locations where fast food is served. Assume the user [fast fod] wanted the correct version of the query when the query contains a simple misspelling. [bus stop] This query requests the locations of bus stops. [coffee shops] This query requests the locations of businesses that sell coffee. [gym] This query requests the locations of fitness centers. [gas San This query requests the locations of gas stations with a location modifier Francisco] of San Francisco. Product and Service Queries A product or service query is a query about something that can be purchased at or is offered by a business or POI. 20  Query Explanation This query requests the locations of businesses that sell a specific type of [vanilla latte] coffee drink. [deep-tissue This query requests the locations of businesses that offer a unique style of massage ] massage. This query requests the locations of businesses that sell pencil, pens, [school supplies] notebooks, and everything else students need for school. This query requests the locations of businesses that offer truck tire repair [truck tire repair] services. This query requests the locations of businesses where ATV tours are [ATV tours] available. Not every hair salon offers perms, but this query requests the locations of [perms] ones that do. [deep dish pizza This query requests the locations of places that sell a specific type of pizza detroit] in a particular city. Coordinate and “My location” queries Some queries are simply a set of coordinates or the words ”my location” (or something similar). Query Explanation [36.082857, The user has specified a location using lat/long coordinates. -115.172916] [my location] The user has typed these (or similar) words to find out where they are. Emoji Queries Some queries are not made using words. Emojis represent the category they picture. Use the most literal meaning. Query Explanation ⛽ The fuel pump represents a search for places to get gas. 🍕 The pizza slice stands for the query “pizza.” ☕ The coffee cup represents the query “coffee.” 21 Queries with no maps intent Some queries have no maps intent. They includes, but are not limited to, queries that: Do not refer to a physical location Do refer to a physical location but have an information intent rather than a location intent Refer to a brand or company with a predominantly online intent, like Groupon or LinkedIn Ask about the time or weather in a specific location Rate all results for these queries Bad. Query Explanation [eureka This is a weather query with no maps intent. Rate all results Bad. temperature] [time in new york This is a “time in” query with no maps intent. Rate all results Bad. city] This is a query about a company that is mainly used online. It does not have maps [facebook] location intent. Rate all results Bad. [is cucumber a fruit This is an information-seeking query that has nothing to do with finding a location. or a vegetable] Rate all results Bad. 2.1.1. Research Expectation Research will help you understand the user’s query and intent: Use a search engine to investigate and understand the query intent Use official resources like businesses’ official websites, national postal service websites, and government websites whenever possible Use your own local knowledge Use the information provided in the rating interface 2.2. Result Types A query can return one of three types of results. Result Type Explanation Example Businesses and POI type results will show a name in the top field, followed by an address and a Business/POI classification. The same rating rules will apply to both Business and POI results. 22  Result Type Explanation Example These results can be one of multiple address types, including residential address or locality. Instead of a business or POI name, this kind of result will show the first line of the address in the top field, followed by the entire address below. This type of result will not Address include a classification. Note: If a result does not include a classification, don’t assume the result type is an address. You must still do research to verify this, as not all POIs will have a classification. The results for POIs or features without an expected address, like bus stops, bridges, or mountains, may sometimes look like address type results. Check to see if the result includes a classification. If it does, the result refers to a feature or POI, not an address. Features At first glance, the result on the without an right might look like it’s for an expected intersection. However, since it address includes a classification, you can see that it actually refers to a POI — in this case, a bus stop. This result for a landmark square might be mistaken for a result referring to a street with the same name. Since it contains a classification, in this case “Park,” you can see that it actually refers to a POI. 2.3. Location Intent A result is considered most relevant when it is located in the user’s area of expected results. There are two types of Location Intent: Explicit: The query is made clearly and indicates a specific location or area where results are expected. 23  Implicit: The location expectation is not as clear, so you must use context clues, in the form of user location and viewport, to discover the area in which the user is expecting results. 2.3.1. Explicit Location When there is an explicit location mentioned in the query, you can ignore the user and viewport locations. The user has told you exactly where they want to find results. If the query contains words like “near me” or “nearest,” the user’s location, not the viewport location, should be considered the explicit location intent. Query Expected results Notes Places to buy this drink [bubble tea tully road san User and viewport locations should be on Tully Road in San jose] ignored. Jose. Museums in the city of User and viewport locations should be [Boston museums] Boston ignored. The queried chain User and viewport locations should be [kfc Philadelphia] business within the ignored. stated locality. Relevant POIs near the Viewport location should be ignored, [food near me] user’s location even if fresh. [nearby/nearest Relevant POIs near the Viewport location should be ignored, starbucks] user’s location even if fresh. Result placed on the Viewport location should be ignored, [my location] user’s location even if fresh. 2.3.2. Implicit Location If the user’s location intent is not expressly stated in the query, for example [chinese] or [479 margarita ave], use the user location, viewport location, and viewport age to determine the area of expected results. Viewport User Location Location Intent When the user is within a fresh viewport, take the user location as location intent. Results are generally expected in Inside Viewport or near the viewport, and results inside the area cannot be rated Bad because of distance alone. Results are expected in or near the viewport area. All relevant Fresh Outside results inside the viewport are eligible for a rating of Viewport Excellent. If no results can be found in or near the viewport, consider the user location a secondary location intent. Missing When the user is missing, the viewport is the location intent. 24  Viewport User Location Location Intent Inside Viewport When the viewport is stale, consider only the user location as Outside location intent. Stale Viewport Use the stale viewport as location intent when the user Missing location is missing. Viewport Age Present or Consider the viewport fresh when the viewport age is missing. Missing Missing The user location sets the location intent when the viewport is Present not present. Missing entirely When the user location and viewport are missing, the test Missing locale becomes the location intent with a strong focus on prominent results. 3.Rating the Query-Level Navigational Result Question Once you have identified user intent, you will need to answer a query-level question. You must decide whether the intent is unique and clear enough that there is only a single (navigational) result in the real world that could fully satisfy it. You will answer this question before you rate any results, since you don’t need to know what the results are in order to learn if there’s a real-world result that could fully satisfy the user’s intent: If there is a real-world navigational result for the query, answer Yes If there are no possible real-world navigational results for the query, answer No You will answer this question once for each query, whether or not any results are shown. 4. Rating Result-Level Issues For each result, you must check to see whether: There is an issue with the language/script of the result name/title The business/POI is closed or does not exist 4.1.Result Name/Title in Unexpected Language or Script A result name or title is in an expected language or script when it is: In the language/script of the test locale In the language/script of the query In a language/script of the result region Any combination of any of the languages or scripts described above 25  An official company, chain, or brand name commonly used in the market, even if this is not any of the languages or scripts described above Below Left: Result name is Atlantis Casino Resort (in red box), address details are below (in blue box). Below Right: Result is an address, so the first line of the address appears in the red box as the Address title. All the address details appear below it in the blue box. When all or part of a result name or title is in an unexpected language or script, users will not understand it. In these cases, select the Result name/title is in unexpected language or script checkbox. No further rating will be required. Note: When the address details, found below the name/title, are in an unexpected language or script, the result can still be understood and rated for relevance and name accuracy. For more on what to do when the address details (not the name or title) are in an unexpected language or script, see Language/Script Issue in Address. Minor Differences in Spelling Don’t consider minor spelling issues, such as missing or added diacritics (e instead of é), as Unexpected Language/Script. This also applies to unnecessary or less relevant parts of a name or title. Use this checkbox only to report issues that interfere with understanding the name/title and issues listed in your Country Specific guidelines. 26 4.1.1. Name of Business/POI Results Pay special attention to the language and script of brand and chain names. See your Country Specific guidelines for more information. Expected Language Explanation Example Expected language/script: Test locale: en_US The query gives clues as to the Result in California, USA language in which the user Query: [mystery] expects results. Any result that Result name: Mystery Spot matches the most specific part of the query or corrects a Result name is in English and satisfies spelling issue in the query is the query. considered to be a result in an Query expected language. This also Unexpected language/script: includes remote results in Test locale: en_US foreign countries. Result in California, USA Query: [market] But: Consider official result name variations and default to Result name: ေစျးကွက် the test locale for very short queries. The result name is “Market” in Burmese script. The language of the test locale Expected language/script: is generally considered an Test locale: es_ES expected language and script. Result in California, USA Query: [college] If the query is so short that the Result name: Colegio Skyline language cannot be identified, assume that the query language The result name is in Spanish, which is that of the test locale. matches the language of the test locale, so this result title is expected and Test Locale When the test locale and the acceptable. and query locale differ, consider the Query Locale query locale an expected Unexpected language/script: language, as it provides more Test locale: en_US detailed information. Result in California, USA In the example below, the Query Query: [college] Locale is set to English in Result name: Colegio Skyline Indonesia. Therefore English is an expected language: “Colegio” is in Spanish, but the test Query Locale: en_ID locale is in English, so the result title is in Test Locale: id_ID an unexpected language. 27 Expected Language Explanation Example Expected language/script: Test locale: en_US Result in Rio de Janeiro, BR Query: [beach] Result name: Praia de Copacabana Result satisfies the category “beach” and the result’s name is in Latin script and the Portuguese language, which is the language of Brazil. The language/script of the Result region result’s region is expected as Unexpected language/script: the result language. Test locale: en_US Result in Rio, BR Query: [beach] Result name: Playa de Copacabana Though the result name is in the same script as the test locale and query language, returning the name in Spanish is unexpected language because the language of Brazil is Portuguese. 28 Expected Language Explanation Example Expected language/script: Test locale: ar_SA Result in Riyadh, SA Query: [‫]ماكدونالدز‬ Result name: McDonald's The name of this restaurant is in the Any official brand, chain , or language of the official brand name. business name in the language of the result’s region or the Expected language/script: official brand name commonly Test locale: ar_SA used in the market, even if it is Result in Riyadh, SA not in the language of the Query: [‫]ماكدونالدز‬ query, test locale, or result Result name: [‫]ماكدونالدز‬ Brand, chain, and region, is expected. business names The name of this restaurant is in the In bilingual areas, official language of the region names in any of the languages of the area are expected, Unexpected language/script: unless otherwise specified in Test locale: ar_SA the Country Specific Result in Riyadh, SA Guidelines. Query: [‫]ماكدونالدز‬ Result name: [マクドナルド] The name of this restaurant is in Japanese, which is not the language of the query, the locale, the result region, or the official brand name. 29 4.1.2. Title of Address Results Unlike business/POI results, address results do not have an actual name. Instead, the first line of the address appears as the title. Rate Result name/title is in unexpected Language or Script when the title (first line of the address) is in an unexpected language or script. Localized and Non-Localized Components Address results often cannot be translated and are accepted in the original version of the result location. Some features, however, are translated into the test locale’s language, especially for bilingual or English-speaking areas. Although the preferred language is the test locale’s language, the query language and language of the result’s location are also accepted. Localized components may include: City State Country Appropriate localization depends on the result itself and the language it is translated into. Research and local knowledge will help you decide if the translation has been done appropriately. Components that are not typically localized include: Feature type (square, bridge, roundabout, or intersection) Street names and street addresses including building names Use the Address Accuracy Incorrect – Language/Script Issue checkbox if you find issues of this nature in any of the address components in the address details (not the address title) of any result. 4.2. Business/POI is Closed or Does not Exist Use this checkbox when, after careful research you determine that a result is: Closed/non-existent Randomly moving (like a food truck without a fixed schedule) A non-recurring past event with no further significance that does not carry any maps intent Important: If you rate a result as Business/POI is closed or does not exist, you must still give a relevance rating as if the place were open or did exist. Remember, this rating can only be applied to business or POI results, not to address type results. 30 Use this checkbox to indicate a POI has closed only if there is evidence to prove the business indicated in the result is no longer operational. When you can’t make a decision about business closure due to lack of resources, assume the business could exist and rate the name, address, and pin accuracy as Can’t Verify (unless there is an obvious data issue like a missing mandatory address component or a pin in the ocean). Note: Businesses and POIs can close for construction, remodeling, vacations, or other reasons. Treat all temporarily closed businesses as if they were open when the closure is announced on the business/POI webpage or managed social media page. There is no limit to how long the closure can be. 4.2.1. Closed/Does Not Exist vs. Inaccurate Name and Address It is important to understand what the result really is before making your rating. If your research reveals that the business has an incorrect name or address, use the drop-down rating options to signal precisely what is wrong. Use the Business/POI is closed or does not exist checkbox only for the specified purposes. See Unclear Results for more guidance and examples. 4.2.2. Rating Relevance of Closed/Non-Existing Business/POI Special consideration is required when rating a closed/non-existing business/POI. Please review section 4.3 carefully with regards to the status and permanently closed vs. open. Never automatically demote to Bad simply because of closure. (You may find other reasons to demote to Bad, but closure should not be the sole reason for the demotion.) 4.3. Business/POI Status is PERMANENT_CLOSURE The following chapter offers a high-level introduction to rating closed/non-existing businesses/POIs in relation to their status (displayed as part of the result information). It addresses Relevance and Data rating. Please also review the more specific guidance in the associated relevance chapter. Note: No matter what is displayed in the status section of the tool, you must always research the business/POI to determine whether it is open or not. This chart shows the possible combinations of status displayed in TryRating plus a result’s actual state based on your research. Real-World State Confirmed by TryRating Status Comments Research Open/Exists None Rate as usual Closed/Does not Exist None Rate as usual 31 Real-World State Confirmed by TryRating Status Comments Research Rate relevance based on Closed/Does not Exist PERMANENT_CLOSURE whether result is expected/ unexpected. See 5.19. Rate relevance based on Open/Exists PERMANENT_CLOSURE whether result is expected/ unexpected. See 5.19. 4.3.1. No Status Shown When no result status shown or the status is blank, you must research to determine whether the business/POI is open or closed/does not exist in the real world. Use the Business/POI is closed or does not exist checkbox when needed. Rate the result relevance as if the business/POI were open or did exist. Use your best judgement to determine how well the result satisfies user intent within the area of expected results. 4.3.2.Status Shown as PERMANENT_CLOSURE When the result status is PERMANENT_CLOSURE, follow the research steps to determine if the business in fact is open in real world or not, and use the Business/POI is closed or does not exist checkbox when needed. The relevance rating for a result with PERMANENT_CLOSURE status depends on whether this result is expected or unexpected based on the specific relevance guidance in 5.19. Rating Relevance when Status is set to PERMANENT_CLOSURE and 10.8. Permanently Closed Results 5. Relevance Relevance for each result is found by assessing the relationship between: Satisfying user intent Prominence Distance Complete each step of this workflow before assigning your relevance rating. General Rating Rules Rate each result individually and demote based on relevance and distance. Always rate against the real world: If there is