Research ethics in language testing

• Researchers often use both qualitative and quantitative research methods across various disciplines.
• There's a divide among researchers, some favoring qualitative, and others quantitative approaches.
• Both qualitative and quantitative methods have their own advantages and disadvantages.

Study Aim

• The study aims to discuss the pros and cons of using both qualitative and quantitative research approaches, especially when it comes to language testing and assessment research.
• Ethical considerations are also discussed within the study.

Key Findings

• Qualitative methods excel at providing insights needed for designing, administering, and interpreting language assessments, and understanding test-taker behavior.
• Drawbacks include smaller sample sizes and the time needed for data collection.
• Quantitative methods use larger samples and require less data collection time.
• Shortcomings include a less in-depth view and potential neglect of test-taker experiences.
• Quantitative research is currently more dominant in language testing and assessment.

Introduction to Research Approaches

• Qualitative and quantitative research approaches are common in education, sociology, psychology, and history.
• "Paradigm wars" exist, with researchers divided between interpretivism and positivism.
• Positivistic researchers believe the social world is concrete and can be objectively measured.
• Interpretive researchers argue reality is socially constructed and understood subjectively.
• Debates exist over whether qualitative or quantitative evidence is superior.

History of Research Paradigms

• The positivist model dominated social and educational research in the early 20th century.
• Standardized tests, systematic observation, experiments, surveys, and statistical analysis were prioritized.
• After the 1960s, new approaches like symbolic interactionist ethnography and critical research emerged.
• It signified a shift of research methods from quantitative to qualitative.
• Qualitative research grew from dissatisfaction with knowledge generation within positivism.
• Ethical considerations were introduced later than research methods and approaches.

Language Testing

• Language testing research has utilized various techniques.
• A "test" demonstrates competence/incompetence and ability/inability.
• Academic tests determine if a student can advance, assess progress, and compare students.
• Tests are tools for public policy, ensuring standards across regions and identifying top performers.
• University admission tests are high-stakes examples.
• In language testing, testers evaluate how accurately a test reflects a candidate's skill in areas like reading, writing, vocabulary, or speaking.
• Language testing is a complex social issue but affects opportunities like promotions, employment, citizenship, etc.
• It also influences curriculum.
• Validity and reliability are vital in language testing.

Qualitative Research Approaches

Defining qualitative research is challenging, as it lacks one theory, paradigm, or set of methods.

• It encompasses various methods across research subjects.
• Qualitative research produces findings without statistical procedures or quantification.
• It focuses on lives, experiences, behaviors, emotions, and cultural phenomena.
• It incorporates multiple realities and involves analysis of subjective meaning, collecting non-standardized data.

Qualitative Research Perspectives

• Qualitative research focuses on how people make sense of the world.
• A multi-method approach involves an interpretive, naturalistic approach to the subject.
• Qualitative research seeks to describe, decode, and translate meanings.
• It addresses phenomena in the social world and encompasses positive and negative perspectives.

Advantages of Qualitative Research

• Qualitative research provides detailed descriptions of participants' feelings, opinions, and experiences.
• It also interprets the meanings of their actions.
• Qualitative results show the relationship of information processing with performance specifically and deeply.
• Qualitative approaches help to achieve deeper insights into designing, administering, and interpreting language assessment.

Interpretivism

• Interpretivism understands the human experience holistically in specific settings.
• Qualitative research crosses disciplines and covers epistemological viewpoints and interpretive techniques.
• Language assessment cannot be separated from context, culture, and values.
• Qualitative research focuses on content-related variables like test-taker characteristics.

Interpretivism Research Approach

• Interpretivism is an ideographic research, which studies individual cases or events.
• It can understand different people's voices, meanings, and events.
• The source of knowledge is the meaning of different events.
• Qualitative techniques analyze candidate, interviewer, and cultural behaviors during speaking tests.
• Qualitative research allows researchers to discover participants’ inner experiences.
• Assessors can understand the meaning of scores or grades.
• Qualitative methods include:
• Participant-observation
• Unstructured interviews
• Direct observation
• Describing records

Qualitative Data Collection

• Researchers directly interact with participants during data collection.
• Data collection is subjective and detailed.
• Deploys face-to-face/telephone interview techniques to collect data and emotions.
• The interactive approach has a flexible structure and produces thorough analyses.
• Participants have freedom to determine what is consistent for them, and complex issues are understood easily.
• Qualitative research captures dynamics and contributes to understanding complex features of assessment.

Disadvantages of Qualitative Research

• Qualitative research may leave out contextual sensitivities, focusing more on meanings and experiences.
• Researchers can experience writing instructors
  • However, the contextual influence on the assessment isn't considered, bypassing cultural impacts.
• Policy-makers may give low credibility to results from the qualitative approach.
  • Examples
  • Stakeholders use quantitative research, or aim to quantify teacher's and student's performance
• Purely qualitative research neglects social and cultural constructions of variables.
• Smaller sample sizes raise generalizability issues.
  • Some have admitted that studies do not wish to claim widers generalisation due to the small sample size

Data Analysis

• Data analysis can be complex, with elusive data and stringent requirements.
• Developing undeveloped questions into researchable forms is harder.
• Refining questions may be continuous throughout a study.
• Analysis of cases takes much time, and generalization happens in a limited way.
• Legislators and policy makers may need faster quantitative results.
• Qualitative research becomes prominent in language testing as accompaniment to quantitative analysis.
• Generalizability isn't a problem if the focus is on the phenomenon.
• Educational research should not regarded as generalizable due to contextual variables.

Example of Contradictory Factors

• In some contexts tests are quantitative which allows the teacher to be condemned if student scores are low.
• However, it fails to consider other circumstances.
• Qualitative researchers can express what works.

Quantitative Definitions

• Emphasizes quantification in data collection and analysis
• Aims to answer questions about quantity or the extent of something.
• Stresses measuring variables.

Numerical Representation

• Aims to identify patterns in human lives, separating the social world into numerical variables.
• Explores associations with statistical techniques.

Focus of Quantitative

• Concerned with aspects of social behavior and quantification, rather than individual interpretation.
• Underpinned by positivism; person and reality are separate.
• Characterized by:
  • Objective reality, statistics, content analysis, data validity, test result reliability.
  • Nomothetic approach with knowledge gained via empirical testing.

Quantitative Research Advantages

• Generalizable to a population due to larger, random samples.
• Employs statistical software (e.g., SPSS) for analysis.
• Data analysis is less time-consuming.
• Based on positivist paradigm; measures variables.

Quantitative Research Disadvantages

• Positivism leaves out social phenomenon meanings and deeper explanations.
• Fails to account for social reality shaping and maintenance.
• Positivism can't account for how people interpret actions.
• It has tendencies of taking snap shots of a phenomenon
• "Positivist" research presents difficulties by not allowing education significance.

Quantitative Method

• Often overlooks respondent perspectives because of limited researchers and particpant interaction.
• Language Assessment has a strong orientation and tradition by employing psychometric procedures
• Focuses on test score validity.
• Testing is accounted for by measurement and psychometric-structural phase with emphasis on test reliability.

Dominant Testing Method

• Language testing is quantitative rather than qualitative in general.
• Language testing uses test validation and quantitative paradigms.
• Researchers around the world employ quantitative techniques for research.

Ethical considerations

• Ethical issues are paramount in research matter due to unbiased selection and fairness
• Actions can be quite damaging resulting in concern for effect on others
• Can result in myriad issues
  • A) Inform "participants" about the research/results
  • B) Maintain result confidentiality is crucial including a promise to maintain

Email risk analysis

• Sending research or confidential by email may result in hacking or forward email
• Confidentiallty is not absolute and can be difficult to control.
• A careful balance must be upheld in language testing that focuses both on professional duty and responsibility.
• Minimizing harm to all involved.

Key Principles

• Respecting autonomy
• Protecting privacy
• Reciprocity, Treating people equitably
• Have respect for dignity
• Establish two codes - A) Code of rights or code of ethics
• B)Code of best practice
• Consider morality vs ideals
• Allowing participants to chose results in subjectiing to harm

Cultural Variance

• Principles/dilemmas are context specific
• Culture can heavily influence.
• Developed countires like UK, US, CA, Aus have systems, not developing countries
• There are not absolute ethical rules or practices.
• Ethical issues with qualitative can be more complex resulting in researcher intimacy

Research paradigms

• Aims to look at strengths and weaknesses in paradigms such as quantitative and qualitative
• Qualitative benefits = Achievng understanding, admin/design, interviewer behaviour
• Sample size, ungenralizable, and policy makers could be negative aspects here. Quantitative is trusty and has larger testing sizes
• However it can be costly, leaves effects of testing.