30 Questions
What is the primary challenge that stakeholders from different technical domains face during requirements elicitation?
Linguistic ambiguity due to terminological discrepancies
What is the potential consequence of not addressing ambiguity during requirements elicitation meetings?
Frustration and distrust, and problems at later stages of development
How does the proposed approach identify ambiguous terms between different domains?
By building domain-specific language models and comparing word embeddings to measure differences in term usage
What is the purpose of ranking ambiguous terms by ambiguity score in the proposed approach?
To estimate the potential ambiguity of a term across the domains of interest
How many potential elicitation scenarios were used to evaluate the proposed approach?
Seven
How many domains were involved in the evaluation of the proposed approach?
Five
What types of terms were found to have a significant impact on the results, and how do they affect the meaning of words in domain-specific corpora?
High-level terms, such as consequence, and named entities, like proper nouns, which do not vary their meaning regardless of their linguistic context.
What are some problems related to the evaluation process of the method that might affect the results?
The interpretation of instructions by the human assessor and their lack of domain knowledge.
Where can the source code of the approach, the evaluation data, and the support algorithms be found?
In the authors' repository.
What are the three main contributions of the current paper compared to the original work?
An improved approach to compute the set of dominant shared terms, a systematic evaluation of the proposed method, and a thorough review of related work.
What is implied by a tie in the ground-truth ranking?
A similar degree of ambiguity for two terms as observed by annotators
What is the structure of the remainder of the paper, and what topics are covered in each section?
The paper is structured into sections: Sect. 2 provides background on ambiguity and word embeddings, Sect. 3 outlines the proposed approach, Sect. 4 presents the research design and results, and Sect. 5 reviews related work.
Why should pairs of terms that are ties in the ground-truth ranking be excluded from the evaluation?
Because the relative order in which the two terms appear in the sample ranking should not be relevant
What is the relationship between the current paper and the AIRE'18 workshop, and what is the reference for the original work?
The current paper extends a contribution to the AIRE'18 workshop, and the original work is referenced as Ferrari et al. (2018a).
How should ties in the sample ranking be handled when ties in the ground-truth ranking are excluded?
As errors from the method, as it has not taken a ranking decision in a case for which a ranking decision exists in the ground-truth
What is the analogy drawn from in the evaluation of translation ranking in machine translation?
The semantic closeness of translation candidates to the original sentence
What variant of the Kendall's Tau measure is adopted for evaluating ranked data with ties?
The Kendall's Tau (τ) formula with penalization of ties produced by the automatic system and exclusion of ties from the ground-truth
Who first proposed the use of the Kendall's Tau measure for evaluating ranked data with ties?
Avramidis (2013)
What is the common meaning of 'consequence' in the three domains of CS, EEN, and MED?
The effect of some phenomenon
What is the difference in the most similar words associated with the term 'consequence' in the three domains?
The most similar words in CS are unintended, circumstance, imposed, situation; in EEN, they are invariance, existence, arises, chaotic; and in MED, they are detrimental, vulnerability, stressor, implication.
What do the lists of 200 words associated with the term 'consequence' in the three domains have in common?
Only 9 words
Why do some terms indicating high-level abstract concepts not change their meaning across different domains?
The meaning of these terms is generally accompanied by different words in different domains.
What is the limitation of the current approach in handling ambiguous terms?
It is not able to automatically discard terms that do not change their meaning across domains.
What is the case of the term 'ion' in the Medical Robot scenario?
It has domain-specific usages that were not noticed by the annotators.
What was the order in which the sentence sets were provided to annotators during the experiment?
random order
What do high ranks in the ambiguity score indicate?
higher chance of ambiguity
What type of ambiguity is exhibited by the term 'argument' in the CS and MED domains?
polysemy
What is the synonym of 'argument' in the MED domain?
dispute
What is the meaning of 'argument' in the CS domain?
logical argument
What is the purpose of reporting the top-20 and bottom-20 in the ranked lists?
to discuss notable cases
Study Notes
Ambiguity Detection in Requirements Engineering
- The paper discusses an NLP approach to identify ambiguous terms between different domains and rank them by ambiguity score.
- The approach is based on building domain-specific language models, comparing word embeddings to measure differences in term usage and estimating potential ambiguity across domains.
Challenges and Limitations
- High-level terms like "consequence" do not vary in meaning regardless of linguistic context, making them difficult to detect as ambiguous.
- Terms with different meanings within domains (e.g., "argument" in CS and MED) can be challenging to identify.
- Problems with evaluation process, including human assessor interpretation and lack of domain knowledge.
Methodology
- The approach involves building domain-specific language models, comparing word embeddings, and computing a set of dominant shared terms.
- The Kendall's Tau (τ) formula with penalization of ties is used to evaluate the method.
Evaluation and Results
- The approach is evaluated on seven potential elicitation scenarios involving five domains.
- The results show that some terms, although indicating high-level abstract concepts, do not change their meaning across domains.
- Undetected domain-specific usages of terms, like "ion" in the Medical Robot scenario, are also identified.
Key Findings
- The term "argument" is found to be highly ambiguous, with different meanings in CS and MED.
- The approach is unable to automatically discard terms with high-level abstract concepts that do not change their meaning across domains.
This quiz is based on a research paper that explores the application of Natural Language Processing (NLP) techniques to detect ambiguity in requirements engineering. It focuses on the challenges of effective communication among stakeholders with diverse backgrounds and skills. Test your understanding of the concepts and techniques presented in the paper.
Make Your Own Quizzes and Flashcards
Convert your notes into interactive study material.
Get started for free
