Exploring the Relationship of a Gamified Assessment with Performance PDF

The Spanish Journal of Psychology (2019), 22, e6, 1–10. © Universidad Complutense de Madrid and Colegio Oficial de Psicólogos de Madrid doi:10.1017/sjp.2019.5 Exploring the Relationship of a Gamified Assessment...

The Spanish Journal of Psychology (2019), 22, e6, 1–10. © Universidad Complutense de Madrid and Colegio Oficial de Psicólogos de Madrid doi:10.1017/sjp.2019.5 Exploring the Relationship of a Gamified Assessment with Performance Ioannis Nikolaou, Konstantina Georgiou and Vasiliki Kotsasarlidou Athens University of Economics and Business (Greece) Abstract. Our study explores the validity of a game-based assessment method assessing candidates’ soft skills. Using self-reported measures of performance, (job performance, Organizational Citizenship Behaviors (OCBs), and Great Point Average (GPA), we examined the criterion-related and incremental validity of a game-based assessment, above and beyond the effect of cognitive ability and personality. Our findings indicate that a game-based assessment mea- suring soft skills (adaptability, flexibility, resilience and decision making) can predict self-reported job and academic performance. Moreover, a game-based assessment can predict academic performance above and beyond personality and cognitive ability tests. The effectiveness of gamification in personnel selection is discussed along with research and practical implications introducing recruiters and HR professionals to an innovative selection technique. Received 30 April 2018; Revised 31 October 2018; Accepted 3 November 2018 Keywords: academic performance, game-based assessments, job performance, selection methods. In order to gain a competitive advantage and make a relationship between a game-based assessment and profit from their activities, organizations need a good performance criteria (e.g., perceived job performance, strategy. But to gain a sustainable competitive advan- Grade Point Average-GPA, perceived Organizational tage, that can last a long time and should not be easily Citizenship Behavior-OCB) to explore its criterion imitated by competitors; organizations must have the related validity. We also explore the extent to which a people resources in place to successfully implement game-based assessment predicts performance beyond the strategy. Along these lines, the need to screen out traditional selection methods (personality measures talented prospective employees possessing the required and cognitive ability). skills to fit the job and meet the performance standards is apparent for every business. Traditional selection Traditional selection tests and performance methods, such as general mental ability and personality Cognitive ability and personality tests are widely used tests, predict job performance to some extent (Ryan & nowadays by organizations in an effort to predict Ployhart, 2014). A number of researchers have recently future work performance. Several studies and meta- suggested that the use of gamification in personnel analyses support not only the validity of cognitive selection, such as game-based assessments, might pre- ability and personality tests but also their effective dict job performance beyond traditional selection combination in predicting job performance (Schmitt, methods (e.g., Armstrong, Landers, & Collmus, 2016; 2014). Cognitive ability tests measure the levels of gen- Fetzer, Mcnamara, & Geimer, 2017). Game-based assess- eral cognitive ability or intelligence, as well as aspects ments is a new assessment method incorporating game of it (e.g., numerical, verbal, abstract, and spatial elements in employee selection and is lately widely ability). Meta-analytic findings indicate that both gen- applied in personnel selection practice, raising ques- eral cognitive ability and specific cognitive abilities tions about its ability to predict job performance. To predict successfully performance and work-related the best of our knowledge, no published empirical outcomes (e.g. Ones, Dilchert, & Viswesvaran, 2012). research has established the effectiveness of game- Moreover, cognitive ability is supported to be the based assessments in the employee selection process. single best predictor of performance at work, as well Our study is designed to examine the potential of a as, of performance outcomes in the majority of job game-based assessment in predicting a number of positions and situations (Schmitt, 2014). As far as performance measures. Specifically, we test the personality is concerned, the most popular personality Correspondence concerning this article should be addressed to How to cite this article: Ioannis Nikolaou. Athens University of Economics and Business. Nikolaou, I., Georgiou, K., & Kotsasarlidou, V. (2018). Exploring Department of Management Science and Technology. 104 34 Athens the relationship of a game-based assessment with performance. (Greece). The Spanish Journal of Psychology, 21. e6. Doi:10.1017/SJP.2019.5 E-mail: [email protected] https://doi.org/10.1017/sjp.2019.5 Published online by Cambridge University Press  2  Nikolaou et al. model is the five-factor model of personality (FFM) assessment’s predictive validity and utility in practise. studied extensively in diverse countries and cultures Moreover, the advent of technology has started to around the world. The predictive validity of at least render traditional selection methods obsolete, paving two key factors of the FFM (especially conscientious- the way for more technologically advanced methods ness but also neuroticism) has been well established capable to reduce the cost of hiring and improve appli- across different job positions and organizations, cant reactions. whereas, meta-analytic findings (Barrick, Mount, & Judge, 2001) have also supported the predicted valid- Game-based assessment methods and performance ity of most personality dimensions of the FFM. Gamification, the application of game-design ele- In the performance domain we often study crite- ments in non-game contexts (Armstrong et al., 2016), rion measures, such as academic attainment and has recently caught the attention of researchers and OCB, apart from job performance. OCBs or extra- practitioners in Work/Organizational Psychology and role performance are defined as the voluntary and Human Resources Management, as a promising tool in non-mandatory employee behaviors that positively employee selection. Employee testing methods have influence organizational effectiveness and contribute started to incorporate game elements and designs to the overall productivity of the organization (Smith, turning into assessments that are likely to be more Organ, & Near, 1983). Both emotional and cognitive intelligence have been found to be related to organiza- fun and attractive to candidates, as well as more diffi- tional citizenship behaviors (e.g., Cote & Miners, 2006). cult to fake (Armstrong et al., 2016). The addition of Whereas, personality traits, such as agreeableness and game elements into the assessments might render the conscientiousness, have been found to predict OCB as assessments more difficult for candidates to decode well (e.g., Chiaburu, Oh, Berry, Li & Gardner, 2011). and identify what the correct answer is, as personality Similarly, academic performance has been found to be traits or intentions and behaviors are assessed indi- significantly predicted by personality and cognitive rectly. For example, in a gamified Situational Judgement ability. Academic performance is usually measured Test (SJT) the clothing of the scenarios and answers with student grades or grade point average-GPA, with game elements might make the desirable behav- which is supported to predict performance at work iors less obvious to candidates and as a result, more (Roth, BeVier, Switzer, & Schippmann, 1996). A number difficult to distort intentionally or unintentionally of meta-analytic studies exploring the relationship what their reactions would be in a given situation as it between personality and academic performance sup- is away from real life situations. ported that agreeableness, conscientiousness and Moreover, building on the concept of “stealth assess- openness to experience, as well as intelligence, predict ment”, Fetzer et al. (2017) highlighted the potential academic performance (Poropat, 2009; Strenze, 2007). of game-based assessments in predicting job perfor- The relationship between cognitive ability and aca- mance. Stealth assessments can accurately and effi- demic performance is also well established (Chamorro- ciently diagnose the level of students’ competencies Premuzic & Furnham, 2008). “Academic performance has by extracting continuously performance data that are been the criterion for validating IQ tests for over a century, gathered during the course of playing/learning (Shute, and one would hardly refer to these tests as “intelligence” Ventura, Bauer, & Zapata-Rivera, 2009). In other words, measures if they did not correlate with academic perfor- stealth assessment is an assessment that is “seamlessly mance” (Chamorro-Premuzic & Furnham, 2008, p. 1597). woven into the fabric of the learning or gaming environment It is worth reporting that both general cognitive ability so that it’s virtually invisible…reducing thus test anxiety and specific cognitive abilities (working memory, pro- while not sacrificing validity and consistency” (Shute, cessing speed, spatial ability) can predict academic 2015, p. 63). Along these lines, a gamified assessment performance whereas, specific cognitive abilities can environment might distract candidates from the fact predict academic performance beyond general cogni- that they are assessed, reducing test anxiety and pro- tive ability (Rohde & Thompson, 2007). moting behaviors that are more likely to appear uncon- To sum up, there is a large body of research which sciously instead of the desirable or socially acceptable indicates general mental ability and personality tests ones. Game engagement and the use of contexts diag- as important predictors of performance. However, nosing how an individual handled a given problem – traditional selection methods, such as personality tests, similar to work-sampling techniques - might lead to predict job performance to some extent, whereas, more robust inferences about performance than tradi- they are prone to faking and social desirability tional selection inventories that rely on self-reported (e.g., Morgeson et al., 2007; Ryan & Ployhart, 2014). measures (Fetzer et al., 2017). Taking into consideration Phenomena, that the application of gamification in all the evidence mentioned above, we aim to explore employee testing might restrain increasing thus the the effectiveness of the game-based assessment method https://doi.org/10.1017/sjp.2019.5 Published online by Cambridge University Press  Game-based Assessment in Selection  3 measuring four soft skills (i.e., resilience, adaptability, (Nelson, 1984). Employees who are capable of effective flexibility, and decision-making) by testing whether its decision-making devote effort to analyze information dimensions are related to performance measures over to better understand a company’s threats, opportu- and above traditional selection measures. nities and options, consult other people and collabo- A major challenge that employers nowadays face rate together in making decisions and act proactively when hiring young graduates is the lack of applicants in getting the things done, enhancing thus, organiza- with the right skills and competencies (Picchi, 2016, tional performance (Miller & Lee, 2001). Whereas, August 31). Among the most desirable soft skills that participation in decision-making leads to positive employers are looking for are adaptability, flexi- outcomes within educational settings, such as OCB bility, decision-making, and resilience (e.g., Gray, 2016; (Somech, 2010). McKinsey & Company, 2017). Resilience, the ability to Taking into consideration all the evidence men- bounce back from adversities (Luthans, 2002), might tioned above, we aim to establish the effectiveness of be vital for both personal and job effectiveness with the gamified selection method that we developed by numerous positive outcomes in work and academic testing whether the gamified SJT dimensions are related settings. For example, resilient individuals are likely to to performance and in particular, to performance have higher levels of job performance, job satisfaction measures, OCB and GPA, over and above traditional and organizational commitment (e.g., Avey, Reichard, selection measures (e.g., personality tests, cognitive Luthans, & Mhatre, 2011), as well as, OCB (Paul, Bamel, ability); therefore, we state the following hypotheses. & Garg, 2016). Moreover, students with higher levels of resilience are likely to demonstrate increased aca- H1: Game-based assessment dimensions will be demic performance levels, as well as higher class positive associated with participants’ job perfor- participation, enjoyment and self-esteem (Martin & mance scores. Marsh, 2006, 2008). Similarly, adaptability, the “response H2: Game-based assessment dimensions will be or people’s adjustment to changing environmental situa- positive associated with participants’ GPA. tions” (Hamtiaux, Houssemand, & Vrignaud, 2013, H3: Game-based assessment dimensions will be p. 130) has positive outcomes in both academic and positive associated with participants’ OCB. work contexts. For example, successful students (GPA H4: Game-based assessment dimensions will provide of 80% or more) were found to have high levels of incremental validity above and beyond the effect of interpersonal, adaptability, and stress management cognitive ability and personality in predicting par- skills (Parker et al., 2004). Moreover, high adaptability ticipants’ job performance scores. is related to positive relationships and behaviors in H5: Game-based assessment dimensions will provide school, such as studying, leadership, and reduced incremental validity above and beyond the effect of school problems (Brackett, Rivers, Reyes, & Salovey, cognitive ability and personality in predicting partici- 2012). In the work context, adaptability is important pants’ GPA. in performing well, handling ambiguity, and dealing H6: Game-based assessment dimensions will provide with uncertainty and stress (Kehoe, 2000). Whereas, incremental validity above and beyond the effect of “volunteering to help co-workers (an aspect of OCB) might cognitive ability and personality in predicting partici- require one to adapt to changing co-worker behaviour” pants’ OCB. (Ployhart & Bliese, 2006, p. 11). Similarly to adapt- ability, flexibility, defined as the individual’s capacity Method to adapt, is likely to have positive outcomes in work, Sample & Procedure academic and job seeking settings (Golden & Powell, 2000). Individuals with high levels of flexibility are The study was conducted in Greece during the last able to address different situations creating thus value months of 2017, attracting participants via the authors’ to organizations instead of harming them because of university career office, along with post-graduate and their inability to adjust in changes (Bhattacharya, final-year undergraduate students or recent graduates. Gibson, & Doty, 2005). Moreover, OCB performers are We contacted final-year undergraduate students, grad- likely to increase their flexibility in order to adjust to uate students or recent graduates to participate in a the requirements of various roles and settings at work survey about a selection method, as these students displaying thus behaviors that contribute to organiza- were approaching graduation and were likely to search tional effectiveness (Kwan & Mao, 2011). Organizational for employment soon (e.g., van Iddekinge, Lanivich, success, especially in changing environments, depends Roth, & Junco, 2016). also largely on effective decision-making, defined as The data collection took place in two phases. In the an intellectual process leading to a response to cir- first phase, participants were invited to complete the cumstances through the selection among alternatives self-reported measures of cognitive ability, personality, https://doi.org/10.1017/sjp.2019.5 Published online by Cambridge University Press  4  Nikolaou et al. performance measures and OCB. Three to four weeks Performance measures. Overall job performance was after completion, participating individuals of the first self-evaluated by working individuals only using a phase were invited to play the game-based assessment. measure used by Nikolaou and Robertson (2001). It 193 participants took part in the first phase and 120 of consists of six items where the individual has to indi- them participated in the second phase, as well, a cate whether she/he agrees or disagrees with the response rate of 62%. The majority of them were behavior described in a five-point scale ranging from 1 females (64%) with a mean age of 26 years. As far as (strongly disagree) to 5 (strongly agree). An overall job their education level is concerned, 46% of the partic- performance score was calculated by averaging the ipants were final year undergraduates, 15% were scores of the six items eliciting internal consistency post-graduate students, another 15% were univer- reliability of.91. Example items include “Achieve the sity graduates and 24% had already acquired a post- objectives of the job” and “Demonstrates expertise in all graduate degree. Most of them (55%) were currently aspects of the job”. We also asked participants to indicate employed, working in entry-level (57.5%) or middle- their GPA from their first degree in order to use it as an level positions (27.5%). alternative to job performance for non-working indi- viduals. The range of the grading system in Greek Measures public universities is 0.00–10.00 (Excellent = 8.50–10.00, Very Good = 6.5–8.49, Good = 5.00 –6.49, and Fail = Cognitive ability. This was measured with items taken 0.00–4.59). The GPA reported by participants was the from the International Cognitive Ability Resource average grade awarded for the duration of their bach- (ICAR) (2014),1. ICAR is a public-domain and open- elor studies. source tool created by Condon and Revelle (2014), aim- Organizational Citizenship Behavior (OCB). OCBs were ing to provide a large and dynamic bank of cognitive self-evaluated by working individuals only using a ability measures for use in a wide variety of applica- measure developed by Smith et al. (1983). It consists of tions, including research. The test includes four item 16 items where the individual has to indicate whether types: Three-Dimensional Rotations, Letter and Number she/he agrees or disagrees with the behavior described Series, Matrix Reasoning, and Verbal Reasoning. We in a five-point scale ranging from 1 (strongly disagree) to used the 11 Matrix Reasoning items, which contain 5 (strongly agree). The original scale measures two sub- stimuli similar to those used in Raven’s Progressive scales; altruism and generalized compliance. However, Matrices, and which is also more closely related to for the purposes of the current study we only used the abstract reasoning. “The stimuli are 3x3 arrays of geo- overall OCB score eliciting internal consistency reli- metric shapes with one of the nine shapes missing. ability of.70. Example items include “I help other Participants are instructed to identify which of six employees with their work when they have been absent” and geometric shapes presented as response choices will “I exhibit punctuality in arriving at work on time in the best complete the stimuli” (ICAR, 2014, p. 2).2 It is morning and after lunch breaks”. worth noting that the correct answer is only one, Soft skills. We used a Game-Based Assessment (GBA) whereas the options “None of the above” and “Do not developed by Owiwi4 in order to measure the four soft know” are also available. An overall score is calcu- skills evaluated by the game, namely resilience, adapt- lated, with high scores indicating higher levels of ability, flexibility and decision-making. The four skills cognitive ability3. are evaluated following a SJT methodology converted Personality. Participants completed the 50 items into an on-line game environment, with fictional char- International Personality Item Pool (IPIP; Goldberg acters. The Owiwi game has demonstrated satisfactory et al., 2006) to assess the Five-Factor model of psychometric elements and increased equivalence personality. Each scale consisted of 10 items. Standard with the originally developed SJT measuring the IPIP instructions were presented to participants, who four soft skills (Georgiou, Nikolaou, & Gouras, 2017). responded on a 5-point Likert-type scale ranging from Resilience is defined as “the developable capacity to 1 (inaccurate) to 5 (accurate). Research has reported rebound or bounce back from adversity, conflict, and failure good internal consistencies for IPIP factors (see, for or even positive events, progress, and increased responsi- example, Lim & Ployhart, 2006). In our study, reli- bility” (Luthans, 2002, p. 702), “Αdaptability is related to ability estimates were.81 for conscientiousness,.83 for change and how people deal with it; that is to say, people’s emotional stability,.83 for extroversion,.79 for agree- adjustment to changing environments” (Hamtiaux et al., ableness, and.75 for openness to experience. 2013, p. 130). Flexibility is defined as the demonstra- tion of “adaptable as opposed to routine behaviors; it is the 1.http://icar-project.com/ 2.https://icar-project.com/ICAR_Catalogue.pdf extent to which employees possess a broad repertoire of 3.For an example item visit https://icar-project.com/ICAR_ Catalogue.pdf 4.www.owiwi.co.uk https://doi.org/10.1017/sjp.2019.5 Published online by Cambridge University Press  Game-based Assessment in Selection  5 behavioral scripts that can be adapted to situation-specific regression analyses for each one of the three criterion demands” (Bhattacharya et al., 2005, p. 624) and finally measures. The results of these analyses are presented decision-making is defined as an intellectual process in Table 2. leading to a response to circumstances through selec- The results of the regression analyses show that flex- tion among alternatives (Nelson, 1984). Individualized ibility and decision-making are positively associated feedback is provided to all participants upon comple- with self-reported job performance and GPA respec- tion of the game. tively. The block of the four skills predict 13%, 7% and 10% of the total variance in job performance, OCB and Results GPA respectively. Therefore, H1 and H2 are partially confirmed, whereas H3 is rejected. Subsequently, we Table 1 presents the inter-correlation matrix of the explored the incremental validity of the game-based study’s variables. An interesting pattern we observe in assessment. In order to explore H4-H6 we conducted a the inter-correlation matrix, is that the cognitive ability number of hierarchical regression analyses, controlling measure is not associated with any of the scales mea- for the effect of cognitive ability and the five-factor sured here. Also, the self-reported job performance model of personality. The results of these analyses are measure is correlated significantly with conscientious- presented in Table 3. ness, emotional stability and openness to experience The results of these analyses demonstrate that the for the five-factor model of personality. Moreover, the soft skills measured by the game-based assessment OCB measure is associated with agreeableness, simi- do not predict additional variance in either job per- larly to past research on the relationships between formance or OCBs for the working individuals of agreeableness and OCB, but not with conscientious- our sample, above the effect of cognitive ability and ness. Finally, the soft skills assessed by the game-based personality rejecting thus H4 and H6. However, they assessment, which is the main focus of the current seem to have an important effect on GPA. More specif- study, are not correlated with any of the criterion mea- ically, both as a group and separately (adaptability and sures, with the exception of the positive correlation decision making) demonstrate a statistical significant between GPA and decision making, rejecting thus H1 relationship with GPA, above and beyond the effect of and H3 and only partially confirming H2. cognitive ability and personality. These results estab- Next, we proceed with the examination of our lish the usefulness of game-based assessments in pre- research hypotheses. Our main focus in this study is dicting educational attainment, as measured by the the suitability of the game-based assessment as a selec- GPA, both as a group and individually in the case of tion tool, above and beyond the well-established effect adaptability and decision making. of cognitive ability and personality, especially conscien- tiousness. Our first three hypotheses deal with the Discussion association between game-based assessment and the three performance criteria. In order to explore these Our study explores the effectiveness of a game-based hypotheses we executed three separated multiple assessment in employee selection. Extending previous Table 1. Inter-Correlation Matrix of Study’s Variables (N = 63–120) Scales Range x SD 1 2 3 4 5 6 7 8 9 10 11 12 13 1. Cognitive ability 11 7.69 2.33 2. Extroversion 36 34.07 7.87 –.03 3. Agreeableness 25 42.05 5.32.08.47** 4. Conscientiousness 35 38.42 7.10 –.05 –.16.00 5. Emotional Stability 33 29.15 7.67.07.20*.14.21* 6. Openness to experience 29 36.78 6.07.40.16.20*.04 –.05 7. Resilience 58 76.35 11.85.10.04.11.14.31.32** 8. Flexibility 58 64.98 12.71.05 –.03.11.07.13 –.02.20* 9. Adaptability 81 74.57 11.60.03.01.07 –.12 –.09.08.40**.26** 10. Decision-making 46 76.42 9.49 –.00.05.12.08.12 –.03.23*.03.20* 11. Job Performance 14 26.21 3.07.13.04.16.40**.26*.32**.13.22 –.07.13 12. OCB 38 64.77 6.78.05.22.39**.14.19.05 –.14 –.03 –.18.12.26* 13. GPA 3.1 7.39 0.72.03 –.06 –.11.13 –.03.00 –.02.08 –.18.25** –.02.07 14. Age 25 26.36 6.21.07 –.18*.03.11 –.04.10.12.19*.11 –.12.22 –.07 –.04 Note: *p <.05. **p <.01. ***p <.001. https://doi.org/10.1017/sjp.2019.5 Published online by Cambridge University Press  6  Nikolaou et al. Table 2. Hierarchical Regression Analysis of the GBA on the Three Criterion Measures Job Performance (N = 63) OCB (n = 63) GPA (N = 113) GBAs Β t ΔR2 ΔF β t ΔR2 ΔF β t ΔR2 ΔF Resilience.14.94.13 2.10 –.12 –.79.07 1.10 –.16 –1.58.10 3.06 Flexibility.30* 2.20.08.58.06.58 Adaptability –.28 –1.85 –.18 1.14.18 1.76 Decision making.17 1.29.20 1.47.25** 2.61 Note: OCB = Organizational Citizenship Behavior; GPA = Great Point Average. *p <.05. **p <.01. ***p <.001. Table 3. Hierarchical Regression Analysis of the GBA on the Three Criterion Measures controlling for Cognitive Ability and Personality Job Performance (N = 63) OCB (n = 63) GPA (N = 113) Predictors β t ΔR2 ΔF β t ΔR2 ΔF β t ΔR2 ΔF Step 1 Cognitive ability.04.30.26 332.**.01.10.20 2.33*.09.92.04.70 Extroversion –.05 –.30.09.55.08.77 Agreeableness.08.53.35* 2.27 –.20 –1.86 Conscientiousness.28* 2.07.18 1.26.18 1.83 Emotional Stability.06.44.08.60 –.07 –.73 Openness to experience.30** 2.44.02.12.02.20 Step 2 Resilience –.02 –.15.06 1.11 –.17 –1.05.03.57 –.20 –1.84.12 3.73** Flexibility.26 1.9 –.03 –.24.07.71 Adaptability –.19 –1.30 –.04 –.23.22* 2.07 Decision making.16 1.15.03.19.26s 2.73 Note: OCB = Organizational Citizenship Behavior; GPA = Great Point Average. *p <.05 **p <.01. ***p <.001. research on Work/Organizational Psychology and tra- Moreover, Armstrong et al. (2016) suggested that ditional selection methods, we introduce a game-based game-based assessments, such as gamified simula- assessment designed to measure candidates’ soft skills tions, might be employed to assess important pre- (e.g., adaptability, flexibility, decision-making) that is dictor constructs like learning agility in employee found to be associated with self-reported measures of selection settings where survey methodology may performance. Our study contributes to employee selec- not be adequate. Along these lines, our study extends tion research, providing some support to the use of research on traditional selection methods, exploring gamification in soft skills assessments and their ability the incremental validity of a game-based assessment to predict performance in work and academic settings. assessing soft skills. Game-based assessments mea- For example, a game-based assessment measuring suring soft skills, such as adaptability and decision soft skills, such as decision-making and flexibility, can making, can predict academic performance (e.g., GPA), predict test-takers’ self-reported job performance and above and beyond traditional selection methods (e.g., GPA. By incorporating game elements into assess- cognitive ability and personality tests). However, the ments that do not use self-reported measures, but soft skills measured by the game-based assessment do assess behavioral intentions, test-takers’ attractive- not predict additional variance in either job perfor- ness and engagement into the assessment might be mance or OCBs, above the effect of cognitive ability enhanced, while it might be more difficult for them to and personality. understand what is being assessed and what the cor- To sum up, both personality and intelligent tests rect answer is (Armstrong et al., 2016; Fetzer et al., have been extensively tested in academic contexts and 2017). As such, the use of game elements and designs their validity in predicting GPA has been established. might improve the validity of assessments. The emergence of internet and technology as well as https://doi.org/10.1017/sjp.2019.5 Published online by Cambridge University Press  Game-based Assessment in Selection  7 the familiarity of new generations with games are measurements has forced many previous studies to likely to reflect an increasing interest in the validity of use self-reported measures of performance (Pransky game-based assessments in predicting academic per- et al., 2006). The use of objective measures or supervi- formance beyond traditional selection methods. The sor’s report of employee’ performance would lead to additive value of using a game-based assessment mea- more robust findings about the predictive validity of suring adaptability and decision making, both as a the game-based assessment. Also, some of the GBA’s group and individually, in predicting GPA beyond dimensions were not found to predict performance. personality (e.g., FFM) and cognitive ability tests (e.g., One reason might be the use of self-reported mea- ICAR), has been established. sures of performance. “It is likely that self-report and Our results are of interest to researchers and prac- objective measures provide information on distinct, dif- titioners of Work/Organizational Psychology interested ferent aspects of work performance. Objective measures, in the prediction of work and academic performance, even in jobs that are apparently routine and straightfor- in that they support the incremental validity of a ward, can present challenging levels of complexity, and game-based assessment over and above traditional may provide an estimation of only one dimension of actual selection methods. They contribute to empirical job performance.” (Pransky et al., 2006, p. 396). Future unknowns about the psychometrics properties and research should explore the ability of the GBA to effectiveness of the use of game-based assessments predict one dimension of performance (e.g., resil- in employee selection. ience or adaptability) using supervisory ratings or Game-based assessments might be used as a sup- objective performance data. plement or replacement tool to traditional selection To establish further the effectiveness of the use of methods as they add to the prediction of perfor- gamification in employee selection, future research mance of candidates or students. However, it is of high should also explore applicants’ reactions. For example, importance to test the effectiveness of game-based assess- candidates perceive multimedia tests as more valid ments using objective measures of performance, such and enjoyable and as a result, they are more satisfied as supervisor’s ratings, and a test-retest reliability with the selection process while organizational attrac- methodology to establish further the psychometric tiveness and positive behavioral intentions are properties of the new assessment method. Moreover, increased (Oostrom, Born, & van der Molen, 2013). The similar to SJTs, game-based assessments might improve impact of game-based assessments on perceived fair- the information gathered about applicants during ness, organizational attractiveness and job pursuit the selection process as well as applicant reactions behaviors should also be investigated to support fur- (Armstrong et al., 2016). Gamification might increase ther their suitability in the selection process. Also, engagement levels which in turn might lead to reten- the current study does not address competence and tion and motivation during the process of selection previous experience with technology, which might as well as better predictions about person-job fit influence test-takers’ performance. For example, (e.g., Chamorro-Premuzic, Akhtar, Winsborough, & candidates who have experience with on-line games Sherman, 2017). Using new technologies and game el- and/or feel competent to use new technology might ements in assessments, recruiters and HR professionals have less anxiety when new technology is used (Cascio might improve selection decisions making more robust & Montealegre, 2016), and as a result, perform better in inferences about their performance as game-based a game-based assessment. In general, the limited assessments do not use self-reported measures that knowledge and lack of empirical research on the use of applicants are likely to fake (Fetzer et al., 2017). gamification in employee selection has made the estab- Another reason that the use of traditional selection lishment of a game-based assessment as an effective methods might be reconsidered and replaced by new selection method even more challenging. game based tools is that the latter are popular among Future research should also explore the role of younger generations. Organizations including game- demographic variables on individuals’ performance in based assessments into the employee selection process game-based assessments. Instead of using demographic might provide a new technologically advanced experi- variables simply as mere control variables in theory ence to applicants sending thus signals about organi- testing, Spector and Brannick (2011) suggest to rethink zational attributes (e.g., innovation) and making the the use of demographics in the first place focusing on the process more fun. mechanisms that explain relations with demographics The present study is not without limitations. First rather than on the demographic variables that serve as of all, performance outcomes were assessed via self- proxies for the real variables of interest. report measures. Although it is suggested that objec- Finally, the study might suffer from common method tive measures are the best indicators of individual variance effects, since we only used self-reported mea- employee performance, the unavailability of such sures. In order to reduce its effect, we asked the https://doi.org/10.1017/sjp.2019.5 Published online by Cambridge University Press  8  Nikolaou et al. participants to complete the measures in two separate Chamorro-Premuzic T., Akhtar R., Winsborough D., & occurrences. Moreover, the Harman’s single factor test Sherman R. A. (2017). The datafication of talent: we conducted following the guidelines of Podsakoff, How technology is advancing the science of human potential at work. Current Opinion in Behavioral Sciences, Mackenzie, Lee, and Podsakoff (2003) discouraged the 18, 13–16. https://doi.org/10.1016/j.cobeha.2017. impact of common method variance on our results. 04.007 Game-based assessments have recently appeared Chamorro-Premuzic T., & Furnham A. (2008). Personality, in employee selection calling for further research on intelligence and approaches to learning as predictors of their validity. Our study contributes to research on academic performance. Personality and Individual employee selection methods by examining the crite- Differences, 44(7), 1596–1603. https://doi.org/10.1016/ rion related validity of a game-based assessment mea- j.paid.2008.01.003 suring soft skills. Findings of our study indicate that Chiabur D. S., Oh I.-S., Berry C. M., Li N., & Gardner R. G. assessments incorporating game elements might pre- (2011). The five-factor model of personality traits and dict self-rated job performance, and academic per- organizational citizenship behaviors: A meta-analysis. formance, as measured by GPA. Moreover, exploring Journal of Applied Psychology, 96, 1140–1166. https://doi. org/10.1037/a0024004 the incremental validity of the game-based assessment Condon D. M., & Revelle W. (2014). The international method, we provided evidence that it can predict GPA cognitive ability resource: Development and initial above and beyond the effect of traditional selection validation of a public-domain measure. Intelligence, methods, such as personality and cognitive ability tests. 43, 52–64. https://doi.org/10.1016/j.intell.2014. These results could change the way organizations and 01.004 colleges approach traditional assessment methods Côte S., & Miners C. T. H. (2006). Emotional intelligence, making the use of gamification in work and academic cognitive intelligence, and job performance. Administrative contexts more widespread in the future. Science Quarterly, 51(1), 1–28. https://doi.org/10.2189/ asqu.51.1.1 Fetzer M., McNamara J., & Geimer J. L. (2017). References Gamification, serious games and personnel selection. In Armstrong M. B., Landers R. N., & Collmus A. B. (2016). H. W. Goldstein, E. D. Pulakos, J. Passmore, & C. Semedo Gamifying recruitment, selection, training, and performance (Eds.), The Wiley Blackwell handbook of the psychology of management: Game-thinking in human resource recruitment, selection and employee retention (pp. 293–309). management. In D. Davis & H. Gangadharbatla (Eds.), West Sussex, UK: John Wiley & Sons Ltd. Emerging research and trends in gamification (pp. 140–165). Georgiou K., Nikolaou I., & Gouras A. (2017). Serious Hershey, PA: IGI Global. gaming in employees’ selection process. In I. Nikolaou, Avey J. B., Reichard R. J., Luthans F., & Mhatre K. H. (2011). Alliance for Organizational Psychology Invited Meta-analysis of the impact of positive psychological Symposium-The impact of technology on recruitment and capital on employee attitudes, behaviors, and selection: An international perspective. Paper presented at performance. Human Resource Development Quarterly, 22(2), the 32nd Annual Conference of the Society for Industrial and 127–152. https://doi.org/10.1002/hrdq.20070 Organizational Psychology, Orlando, USA. Barrick M. R., Mount M. K., & Judge T. A. (2001). Goldberg L. R., Johnson J. A., Eber H. W., Hogan R., Personality and performance at the beginning of the Ashton M. C., Cloninger C. R., & Gough H. G. (2006). new millennium: What do we know and where do The international personality item pool and the future of we go next? International Journal of Selection and public-domain personality measures. Journal of Research in Assessment, 9(1-2), 9–30. https://doi.org/10.1111/1468- Personality, 40(1), 84–96. https://doi.org/10.1016/j.jrp. 2389.00160 2005.08.007 Bhattacharya M., Gibson D. E., & Doty D. H. (2005). The Golden W., & Powell P. (2000). Towards a definition effects of flexibility in employee skills, employee of flexibility: In search of the Holy Grail? Omega, 28(4), behaviors, and human resource practices on firm 373–384. https://doi.org/10.1016/S0305-0483(99) performance. Journal of Management, 31(4), 622–640. 00057-2 https://doi.org/10.1177/0149206304272347 Gray A., (2016). The 10 skills you need to thrive in the Fourth Brackett M. A., Rivers S. E., Reyes M. R., & Salovey P. Industrial Revolution. Retrieved from The World Economic (2012). Enhancing academic performance and social and Forum website: https://www.weforum.org/agenda/ emotional competence with the RULER feeling words 2016/01/the-10-skills-you-need-to-thrive-in-the-fourth- curriculum. Learning and Individual Differences, 22(2), industrial-revolution 218–224. https://doi.org/10.1016/j.lindif.2010. Hamtiaux A., Houssemand C., & Vrignaud P. (2013). 10.002 Individual and career adaptability: Comparing models Cascio W. F., & Montealegre R. (2016). How technology is and measures. Journal of Vocational Behavior, 83(2), 130–141. changing work and organizations. Annual Review of https://doi.org/10.1016/j.jvb.2013.03.006 Organizational Psychology and Organizational Behavior, 3(1), International Cognitive Ability Resource (ICAR) (2014). 349–375. https://doi.org/10.1146/annurev-orgpsych- [Public-domain assessment tool]. Retrieved from https:// 041015-062352 icar-project.com/ https://doi.org/10.1017/sjp.2019.5 Published online by Cambridge University Press  Game-based Assessment in Selection  9 Kehoe J. F. (2000). Managing selection in changing commitment. Vikalpa, 41(4), 308–324. https://doi. organizations: Human resource strategies. San Francisco, org/10.1177/0256090916672765 CA: Jossey-Bass Publ. Picchi A. (2016, August 31). Do you have the “soft skills” Kwan H.-K., & Mao Y. (2011). The role of citizenship employers badly need? CBSNews.com. Retrieved from behavior in personal learning and work–family https://www.cbsnews.com/news/do-you-have-the-soft- enrichment. Frontiers of Business Research in China, 5(1), skills-employers-badly-need/ 96–120. https://doi.org/10.1007/s11782-011-0123-6 Ployhart R. E., & Bliese P. D. (2006). Individual adaptability Lim B.-C., & Ployhart R. E. (2006). Assessing the convergent (I–ADAPT) theory: Conceptualizing the antecedents, and discriminant validity of Goldberg’s international consequences, and measurement of individual differences personality item pool: A multitrait-multimethod in adaptability. In C. Shawn Burke, Linda G. Pierce, & examination. Organizational Research Methods, 9(1), 29–54. Eduardo Salas (Eds.) Advances in human performance and https://doi.org/10.1177/1094428105283193 cognitive engineering research (Vol. 6, 3–39). Bingley, UK: Luthans F. (2002). The need for and meaning of positive Emerald Group Publishing Limited. organizational behavior. Journal of Organizational Behavior, Podsakoff P. M., MacKenzie S. B., Lee J.-Y., & Podsakoff 23(6), 695–706. https://doi.org/10.1002/job.165 N. P. (2003). Common method biases in behavioral research: Martin A. J., & Marsh H. W. (2006). Academic resilience and A critical review of the literature and recommended its psychological and educational correlates: A construct remedies. Journal of Applied Psychology, 88(5), 879–903. validity approach. Psychology in the Schools, 43(3), 267–281. https://doi.org/10.1037/0021-9010.88.5.879 https://doi.org/10.1002/pits.20149 Poropat A. E. (2009). A meta-analysis of the five-factor model Martin A. J., & Marsh H. W. (2008). Academic buoyancy: of personality and academic performance. Psychological Towards an understanding of students’ everyday Bulletin, 135(2), 322–338. https://doi.org/10.1037/a0014996 academic resilience. Journal of School Psychology, 46(1), Pransky G., Finkelstein S., Berndt E., Kyle M., Mackell J., & 53–83. https://doi.org/10.1016/j.jsp.2007.01.002 Tortorice D. (2006). Objective and self-report work McKinsey & Company (Producer). (2017). The digital performance measures: A comparative analysis. future of work: What skills will be needed? [Video]. International Journal of Productivity and Performance Available from https://www.youtube.com/watch?v= Management, 55(5), 390–399. https://doi.org/10.1108/ UV46n44jnoA 17410400610671426 Miller D., & Lee J. (2001). The people make the process: Rohde T. E., & Thompson L. A. (2007). Predicting Commitment to employees, decision making, and academic achievement with cognitive ability. performance. Journal of Management, 27(2), 163–189. Intelligence, 35(1), 83–92. https://doi.org/10.1016/ https://doi.org/10.1177%2F014920630102700203 j.intell.2006.05.004 Morgeson F. P., Campion M. A., Dipboye R. L., Roth P. L., BeVier C. A., Switzer F. S., III., & Schippmann J. S. Hollenbeck J. R., Murphy K., & Schmitt N. (2007). (1996). Meta-analyzing the relationship between grades and Are we getting fooled again? Coming to terms with job performance. Journal of Applied Psychology, 81(5), 548–556. limitations in the use of personality tests for personnel https://doi.org/10.1037/0021-9010.81.5.548 selection. Personnel Psychology, 60(4), 1029–1049. Ryan A. M., & Ployhart R. E. (2014). A century of selection. https://doi.org/10.1111/j.1744-6570.2007.00100.x Annual Review of Psychology, 65(1), 693–717. https://doi. Nelson G. D. (1984). Assessment of health decision making org/10.1146/annurev-psych-010213-115134 skills of adolescents. Retrieved from ERIC database Schmitt N. (2014). Personality and cognitive ability as (ED252774). predictors of effective performance at work. Annual Review Nikolaou I., & Robertson I. T., IV. (2001). The Five-Factor of Organizational Psychology and Organizational Behavior, model of personality and work behavior in Greece. 1(1), 45–65. https://doi.org/10.1146/annurev-orgpsych- European Journal of Work and Organizational Psychology, 10(2), 031413-091255 161–186. https://doi.org/10.1080/13594320143000618 Shute V. (2015, August). Stealth assessment in video Ones D. S., Dilchert S., & Viswesvaran C. (2012). Cognitive games. Paper presented at the Australian Council for abilities. In N. Schmitt (Ed.), The Oxford handbook of Educational Research Research “Conference Learning personnel assessment and selection (pp. 179–224). New York, Assessments: Designing the future conference”. Melbourne, NY: Oxford University Press. Australia. Oostrom J. K., Born M. P., & van der Molen H. T. Shute V. J., Ventura M., Bauer M., & Zapata-Rivera D. (2013). Webcam tests in personnel selection. In D. Derks & (2009). Melding the power of serious games and A. Bakker (Eds.), The psychology of digital media at work embedded assessment to monitor and foster learning. In (pp. 166–180). USA & Canada: Psychology Press. U. Ritterfeld, M. Cody, & P. Vordered (Eds.), Serious games: Parker J. D. A., Creque R. E., Barnhart D. L., Harris J. I., Mechanisms and effects (pp. 295–321). New York and Majeski S. A., Wood L. M.,... Hogan M. J. (2004). London: Routledge, Taylor & Francis. Academic achievement in high school: Does emotional Smith C. A., Organ D. W., & Near J. P. (1983). intelligence matter? Personality and Individual Differences, Organizational citizenship behavior: Its nature and 37(7), 1321–1330. https://doi.org/10.1016/j.paid. antecedents. Journal of Applied Psychology, 68(4), 653–663. 2004.01.002 https://psycnet.apa.org/doi/10.1037/0021-9010.68.4.653 Paul H., Bamel U. K., & Garg P. (2016). Employee Somech A. (2010). Participative decision making resilience and OCB: Mediating effects of organizational in schools: A mediating-moderating analytical https://doi.org/10.1017/sjp.2019.5 Published online by Cambridge University Press  10  Nikolaou et al. framework for understanding school and teacher Strenze T. (2007). Intelligence and socioeconomic success: outcomes. Educational Administration Quarterly, A meta-analytic review of longitudinal research. Intelligence, 46(2), 174–209. https://doi.org/10.1177/ 35(5), 401–426. https://doi.org/10.1016/j.intell.2006.09.004 1094670510361745 van Iddekinge C. H., Lanivich S. E., Roth P. L., & Junco E. Spector P. E., & Brannick M. T. (2011). Methodological (2016). Social media for selection? Validity and adverse urban legends: The misuse of statistical control impact potential of a Facebook-based assessment. Journal variables. Organizational Research Methods, 14(2), of Management, 42(7), 1811–1835. https://doi.org/ 287–305. https://doi.org/10.1177/1094428110369842 10.1177/0149206313515524 https://doi.org/10.1017/sjp.2019.5 Published online by Cambridge University Press