Evaluation of Mathematical Game Design Skills of Pre-Service Classroom Teachers

The purpose of the research is to evaluate the games prepared by pre-service classroom teachers within the scope of “Mathematics Teaching 1” and “Mathematics Teaching 2” courses, which are included in the undergraduate classroom teaching programs in Turkey, and to make predictions on the game design skills of pre-service teachers through them. The study has been conducted as a case study which is one of the qualitative research designs. The sampling group has been composed of 108 mathematical games designed by 386 undergraduate students, who were third year students at Necmettin Erbakan University, department of classroom teaching in 2015-2016 academic year, fall and spring semesters. Data collected through interviews and observations have been analysed with the help of descriptive analysis method. Results indicate that pre-service teachers failed at designing games that are satisfactory in terms of “Game Rules”, “Goals and objectives”, “Outcomes and feedback” and “Interaction”. This can be taken as one of the main indicators of general failure in game designs. According to the other findings, game designs developed by pre-service teachers were partially or completely satisfactory in terms of “Conflict (Competition, challenge, and opposition)” and “Story”


Introduction
Games can be defined as activities that are done voluntarily and freely, provide a source of happiness, stimulate all developmental aspects of children and improve not only skills but also emotions (Razon, 1985).Additionally, the concept of game is taken from many different aspects and defined differently in the literature.Among these, Vygotsky (1967) states that game is a mechanism used by individuals to cope with the environment, and it forms the basis of development during pre-school period.According to him, abstract thought developed with games has a very important place in the development of individuals.According to Piaget (1962), every act of intelligence is defined by the balance between two opposite tendencies; assimilation and accommodation.In assimilation, individuals involve events, objects and cases within the existing ways of thinking that form the organizational mental structures.In accommodation, existing ways of thinking are combined with new aspects of external environment and reorganized.With the acts of intelligence, the individuals accommodate with the requirements of external reality and at the same time preserve their mental structures completely.On the contrary, game is defined with the superiority of assimilation over accommodation.Individuals involve events and objects within their existing mental structures.In other words, Piaget (1962) considers game as an "assimilation function".Accordingly, the studies on game in the literature mostly focus on cognitive development.This is duty the fact that game is defined as providing children with special and extraordinary environments, such as learning, enlightenment and developing meaning (Wood & Attfield, 2005).
In agreement with the conceptual framework provided above, and with the changes and improvements in the structures, rules and purposes of the games in line with the cognitive development of especially the pre-school and primary school children, games are today used as a method of teaching.Findings of many recent studies have showed that educational games are activities that have an important effect on the learning process (Shin, Sutherland, Norris, & Soloway, 2012;Virvou, Katsionis, & Manos, 2005).The common main finding of these studies is that games enable students do "a mental training".In this sense, students participating in games constantly encounter with decision-making cases during the game, and they use problem solving strategies for complex tasks involving interwoven sub-tasks (Johnson, 2005).Gee (2003), defines these complex tasks and the cases students face within these tasks in four steps for video games: (1) engage/probe: gamers should engage in the existing situation and probe this; (2) Hypothesize: they should form a hypothesis by reflecting the experiences acquired through probing; (3) re-probe: they should reprobe using the hypothesis they formed mentally; (4) rethink: they should rethink the original hypothesis evaluating the feedbacks form the game environment.
Similarly, Garris, Ahlers, and Driskell (2002) state that educational games enable students encounter the processes of repetitive decision-making, acting and getting feedback.Additionally, the related literature includes many studies reporting that educational games are very important in developing decision-making, design, strategy, cooperation and problem solving skills of students, and accordingly they are valuable tools for learning and teaching processes (McFarlane, Sparrowhawk, & Heald, 2002;Cohen Group Report, 2011).
Taken the educational benefits mentioned above into consideration, games, which are of increasing and enriching learning quality, should be placed at the basis of the educational programs so that they can provide more benefits for the children.Educators should utilize these features of the games as they plan in-class activities (Morrison, 1998).Many studies on the subjects emphasize the importance of comprehending game world, imaginary world and real world of children for educators.Additionally, children can develop their skills with various knowledge and experiences by combining their experiences in these different worlds only with their educators' awareness and knowledge in the field their abilities to design effective games (Wood & Attfield, 2005;Ceglowsky, 1997).
The models developed by Holland et al. (2003), Moreno-Ger et al. (2008) and Whitton (2012) define the dimensions of effective game design as: (1) Evaluation of game; (2) Cost-effectiveness; (3) Pedagogical effectiveness; (4) Gamification.Shuteve Ke (2012) proposed that core-elements of well-designed games must include: ( 1 2010) also listed 12 game elements from the literature and grouped them into four dimensions, which are shown in Figure 1.Some of the game elements appearing in the four dimensions are also mentioned in the core elements of a well-designed game and elements of an effective game (De Freitas & Jarvis, 2009;De Freitas & Oliver, 2006 Similarly, the model of Prensky (2003) includes 6 key elements of game: (1) Rules: Rules define the borders of the game and provides us with some ways to reach our goals; (2) Goals and objective: goals and objectives provide gamers with a sense of duty and enable them play the game voluntarily and spend time and effort on it; (3) Outcomes and feedback: how the user proceeds towards the goal is notifies through feedbacks.Feedback is provided when something changes due to something user does within game; (4) Conflict: conflict, competition, challenge or opposition is the problem of the game that needs to be solves.The user can feel the senses of fear and excitement within the game as in real life, without encountering real danger, which motivates the user to continue playing and complete the game; (5) Interaction: is a social situation formed between gamers as they play the game; (6) Representation or story: is what the game is about.The story of the game can be presented at the beginning of the game or can be presented in more detail within the game.

Objective
A general evaluation of the information presented above shows that the effective use of games as an educational tool or teaching method especially focusing on cognitive development is very important and game design must be taken systematically for maximum educational benefit.To this end, presenting the game design competitions of primarily teachers and therefore the pre-service teachers is important.Most of the studies is the related literature focus on how learning is realized through games and the effects of the games on the learners (Garris, Ahlers, & Driskell, 2002;Gee, 2003;Kasvi, 2000;Pivec & Kearney, 2007).At this point, studies on educational game design are more important.Accordingly, the purpose of the present research is evaluating the games prepared by pre-service classroom teachers within the scope of "Mathematics Teaching 1 and Mathematics Teaching 2" courses, which are included in the undergraduate classroom teaching programs in Turkey, and making predictions on the game design skills of pre-service teachers through these.
In accordance with this purpose, the problem statement of the research is "What is the quality of the pre-service classroom teachers' game design skills within the framework of games they designed for the requirements of Mathematics Teaching 1 and 2 courses?"Answers to the following questions are sought in order to find an answer to this problem.

Research Design
The present research on defining the quality of the game designs of pre-service classroom teachers is modelled as a case study.Case study can be defined as a research method that investigates phenomena in their own environment, with no distinct borders between the phenomena and their environment, based on "how" and "why" question, and enables the researchers to investigate in detail any phenomenon or event they cannot control (Yıldırım & Şimşek, 2005).What distinguishes case study from other methods is that it is based on 'how' and 'why', enables researchers to be included in a phenomenon or event they cannot control and investigate it in detail (Ekiz, 2009;Yıldırım & Şimşek, 2005).The reasons for using this model in the present research are, that the game designs of pre-service classroom teachers are accepted as a case with no distinct borders and that cannot be controlled by the researchers; and that this model enables researchers get involved in this case, investigate it in detail and find answers to "why-how" questions.
The present study is designed in accordance with "holistic single case study", which is one of the case study designs.Holistic single case studies involve one single analysis unit.It is used to confirm or reject a well-formulated hypothesis, to study unique or extreme cases, or for cases never studies or revealed before (Yıldırım & Şimşek, 2005).The single case studied in the present study is the game designs of pre-service teachers.The present study takes pre-service teachers as an analysis unit.

Participants
The present research is structured to analyse 108 mathematical games designed by 386 students, who were third year students at Necmettin Erbakan University, Department of Classroom Teaching in 2015-2016 academic year fall and spring semesters.The sample was formed in accordance with criterion sampling, which is a purposive sampling method.Purposive sampling enables the studying of cases, which are thought to have rich information (Patton, 1997).The main principle of criterion sampling is studying of the all cases meeting a series of pre-determined criteria.The criterion or criteria can be formed by the researchers of they can use a prepared criterion list (Yıldırım & Şimşek, 2005).The criterion for the selection of the pre-service teachers to participate in the present study was that they were informed of Turkish language teaching technique in various courses (Teaching Principles and Methods, Development and Learning, etc.) and they had practice in these subjects, they had completed their teaching practice.

Data Collection Tool
Data for the present research were collected via observation and interview techniques.Observations provide information on how the incidents actually happen (Çepni, 2007).The purpose of a simple observation is listening to what people say, monitoring what they do and sometimes asking some questions to provide clarity for some questions (Gillham, 2000).Observation technique is utilized to define an incident in detail (Yıldırım & Şimşek, 2005).
"Semi-structured game design observation form" used during the observations conducted for the present research was developed by researchers in accordance with the models provided in the related literature (Prensky, 2003).This form, includes the (1) Rules; (2) Goals and objective; (3) Outcomes and feedback; (4) Conflict (Competition, challenge, opposition); ( 5) Interaction; (6) Representation or story dimensions that define the structure of educational game according to the literature.In order to support observation findings, interviews are frequently utilized (Yıldırım & Şimşek, 2005).Therefore, unstructured interview questions were frequently used for data reliability.

Data Collection Process
The present research studies the games designed by 3 rd year undergraduate classroom teaching students within the scope of "Mathematics Teaching 1 and Mathematics Teaching 2" courses.Observations and interviews made for this purpose have been conducted at the classrooms in the university building where the specified courses have been taught.At each session which the students presented their game designs, at least, two of the researchers have been provided to be observers.All of the students' applications related to the game designs have been recorded on video.Each student group's game design presentations lasted approximately 25 minutes.After the presentation of the game designs of the students, in order to be able to analyse the game design in more detail, questions have been asked by the researchers who follow the presentation.With this practice, it is aimed to support the observations realized by the interviews.
In the data collection process of the research the following steps were followed accordingly.
1.The participating students were informed about the theoretical framework related to the educational game concept.
2. Mathematics subjects related to the learning domains included in the primary school mathematics curriculum were distributed among students.
3. Students were provided with enough time for preparation (3 weeks).
4. Students were asked to organize the game play process for the games they designed in the classroom environment following the following steps.4.3 Evaluation of the game: After playing the game, students were asked to evaluate the game.During the evaluation, interviews were conducted by asking students some questions about the game design.

Data Analysis
Observations conducted for especially qualitative researches can be recorded with video, audio recorders or by taking notes (Yıldırım & Şimşek, 2005).Semistructured observations conducted for the present research were video-recorded by the researchers.During observations, interviews were frequently utilized.While analysing the observation data, all researchers rewatched the game application, which were videorecorded by the researchers, and each researcher marked the related parts on the observation form to show their evaluations of the games and provided explanations.In other words, each researcher individually filled in the semi-structured observation form including the observation levels of criteria required by the game design dimension.After that, the researchers compared their forms together, discussed on each and made one form for each game when they had a consensus.In this process, the grading expressions of specialists for each participant has been compared with each other and reliability has been calculated using the formula of Miles and Huberman (1994) (Reliability=opinion association/ opinion association+opinion separation X100).With the help of the formula, it has been calculated that the correspondence among encoders in the range of 87.25 -100%.Observation data obtained this way are presented in detail in findings part.Data collected through observations are coded and presented in tables, preservice teachers' opinions are presented when necessary to support the observation findings and necessary explanations are provided under the tables.

Findings
The findings of the analyses conducted to evaluate the games designed by pre-service teachers in terms of game rules are presented in Table 1.As presented in Table 1, almost half of the total of 108 games designed for all four grades of primary school are unsatisfactory in terms of rules (1 st grade: 41.67%; 2 nd grade: 51.73%; 3 rd grade: 56.67%; 4 th grade: 52%).
According to the findings games designed by pre-service teachers were unsatisfactory in terms of defining the borders of the game and providing ways of achieving the goal.The rules set by the pre-service teachers mostly failed to provide an effective and uninterrupted game play.The reasons for this failure might be that pre-service teacher failed to predict the possible situations that can occur within the game and they didn't include any rules related to this situation.Some of the statements supporting this finding obtained through interviews conducted during the evaluation of the game are as follows: "The game was supposed to be fun.But an unexpected situation occurred.That's why the game stopped.""I couldn't predict …'s move in the game.I tried to come up with a new rule when … did that move but I guess my friends got a little bored meanwhile.""The ones answering the difficult questions during the game were going to get higher scores.However, questions were not difficult but easy I think.That's why the game took to short.I should have thought of what to do when the game was over." The findings of the analyses conducted to evaluate the games designed by pre-service teachers in terms of goals and objectives are presented in Table 2.As presented in Table 2, almost half of the total of 108 games designed for all four grades of primary school are unsatisfactory in terms of "goals and objectives" (1 st grade: 54.17%; 2 nd grade: 51.72%; 3 rd grade: 46.8%; 4 th grade: 48%).According to the findings games designed by pre-service teachers were unsatisfactory in terms of defining the learning objectives and/or reflecting these on the game and creating a sense of duty and voluntariness.This seriously affected the participation in the game.The goals set for the games failed at providing motivation for the students to participate in the game.The reasons for this failure might be that pre-service teacher failed to set goals appropriate for the class level.Some of the statements supporting this finding obtained through interviews conducted during the evaluation of the game are as follows: "It was fun when we played this game before.Yet, I think it didn't attract attention of my friends here.Nobody wanted to play.And we failed when we forced participation." "The game started well.But later people got bored.When they got bored they played involuntarily.What could I do to prevent this?I think I should have new goals within the game while I designed the game." "I think the game reached its objective.My topic was addition and we solved many operations.I think the only problem was that the students were university students.It would be much better if I did that with 4 th graders." The findings of the analyses conducted to evaluate the games designed by pre-service teachers in terms of outcomes and feedback are presented in Table 3.As presented in Table 3, most of the total of 108 games designed for all four grades of primary school is unsatisfactory in terms of "outcomes and feedbacks".According to the findings experiences acquired in the games designed by pre-service teachers were unsatisfactory in terms of developing learning experiences and providing feedbacks in time and correctly.This can be another source of the general failure of games.First of all, most of the games were away from learning objectives.Additionally, frequent and timeless repetition of feedbacks seriously affected the fluency of the games.The reason for frequent repetition of feedbacks can be that the games couldn't be explained correctly to the players.Some of the statements supporting this finding obtained through interviews conducted during the evaluation of the game are as follows: "I thought I should have intervened when the game was misunderstood.It would be much better if I had provided some information at the beginning." "The game got so far away from my design.So, I had to stop the game and tell them what they should do." "I think the game failed at focusing on the math subject.My friends seemed to like running better." The findings of the analyses conducted to evaluate the games designed by pre-service teachers in terms of "Conflict (Competition, challenge, opposition)" are presented in Table 4.As presented in Table 4, most of the total of 108 games designed for all four grades of primary school is partially or completely satisfactory in terms of "conflict".According to the findings experiences acquired in the games designed by pre-service teachers were satisfactory in terms of providing players with experiences of conflict, competition, challenge and opposition.This can be considered as the main factor providing playability in spite of the general failure resulting from rules, objectives, motivation and feedbacks.Pre-service teachers could manage to include especially the competition factor.Those who didn't stated that they didn't include the competition factor as they thought it is not suitable for primary school children.Additionally, some, who did, failed to adjust the level of competition.
Some of the statements supporting this finding obtained through interviews conducted during the evaluation of the game are as follows: "I didn't want competition in my game design because I didn't want the children to play competitively." "I thought that if my game design included competition, more people would participate, but I think I overdid it.Element related to course content should have been more prominent." "I think this kind of competitive games is not appropriate for children.We shouldn't raise humans who just enjoy winning.Human beings should also enjoy the activity they are in." The findings of the analyses conducted to evaluate the games designed by pre-service teachers in terms of "Interaction" are presented in Table 5.As presented in Table 5, most of the total of 108 games designed for all four grades of primary school is unsatisfactory in terms of "interaction".According to the findings, games designed by pre-service teachers were unsatisfactory in terms of providing social interaction between group members.This failure can be resulted from lack of motivation, or the problem in rules.Some of the statements supporting this finding obtained through interviews conducted during the evaluation of the game are as follows: "My game design provided opportunities for the participation of all class.But most of my friends didn't participate.I think failed to attract their interest." "One member of each team would come to the board.After that, they would replace with other members of their teams.I think the ones on their seats got bored.When nobody wanted to come to the board, I completed the game with the ones who did." The findings of the analyses conducted to evaluate the games designed by pre-service teachers in terms of "Story" are presented in Table 6.As presented in Table 6, most of the total of 108 games designed for all four grades of primary school is partially or completely satisfactory in terms of "story".According to the findings story element the games designed by preservice teachers were satisfactory in terms of including features related to the development level of students.Pre-service teachers paid attention to the stories of their games and considered this as the most important element of their designs.Some of the statements supporting this finding obtained through interviews conducted during the evaluation of the game are as follows: "I think story is the most important element of the game.
Stories should be appropriate for the ages of the children.In other words, children should live the story." "The story setup of what is done in the game should be good.This provides more participation.They love the game more." "The topics should be popular among children.Characters in the topics are also important.Children today might not love the heroes of our childhood.We should provide an environment, which children can love with contemporary characters." The findings of the analyses conducted to evaluate the games designed by pre-service teachers in terms of "Representation" are presented in Table 7.As presented in Table 6, most of the total of 108 games designed for all four grades of primary school is unsatisfactory in terms of "representation".According to the findings, games designed by pre-service teachers were unsatisfactory in terms of representing the learning objectives by their characters, environment, roles, rules and other game elements.This shows that game designs failed to reach its educational objectives.Some of the statements supporting this finding obtained through interviews conducted during the evaluation of the game are as follows: "The game was designed for 3 rd grade geometry learning domain.But I think it included few math related elements." "I thought students would understand the subject better after playing the game, but it didn't go as I expected.It was more fun than educational." "I think my game works more as a practice or review than teaching new topics or raising awareness on something new."

Results and Discussion
The first finding of the research was that, pre-service teachers failed at designing games that are satisfactory in terms of rules.This can be taken as one of the main indicators of general failure in game designs.According to Dondlinger (2007) game rules are one of the elements of effective design.Many studies in the related literature emphasize the importance of rules in an effective design (Waraich, 2004;Zagal, Nussbaum, & Rosas, 2000).Especially in experimental game models formed to gather game design elements in accordance with educational theories underline that games provide a meaningful environment for problem-based learning and games should provide students with environments to discover new ideas and rules instead memorizing things (Kiili, 2005).The finding related to pre-service teachers' failing at defining game rules correctly indicates that the games designed by pre-service teachers will fail to reach at the educational objectives mentioned above.
Another finding of the present research is that, games designed by pre-service teachers were unsatisfactory in terms of "goals and objectives" as well.This negatively affected the continuity of the game and prevented obtaining the expected educational benefits from the games.Swartout and van Lent (2003), state that goals of different levels motivate learners to continue the game.This is considered as the main problem of the games studied within the scope of the present research.Preservice teachers set simple goals, which they thought as appropriate for the developmental levels of primary school children.Yet, game levels changing in accordance with the levels of each learner and their success within the game increase the functionality of games (Gee, 2003).Appropriateness of the goals affects gamers' levels of commitment to the game.Another factor affecting commitment levels is motivation.Waraich (2004) reports that context should be of appropriate quality for meaningful learning and motivation is necessary for learning performance.The positive effects of correctly set goals on motivation have been proven by many researchers before (Amory, Naicker, Vincent, & Adams, 1999;Denis & Jouvelot, 2005;Jennings, 2001).
Third finding of the present research was that game designs of pre-service teachers were also unsatisfactory in terms of "outcomes and feedback".Because pre-service teachers couldn't explain the rules and goals of their games correctly, they had to stop the game and give feedback for correcting the acts that were against rules.Providing a certain level of interaction and feedback is a factor enabling players to carry on with the next goal and finally completely the game in victory.Yet, too much feedback within the game affects the players negatively (Song & Zhang, 2008).The related literature takes feedback in game design in two dimensions; player feedback and system feedback (Van Staalduinen & De Freitas, 2010).Appropriated briefing and system feedback will be given by the game according to the users' achievements to assist the students learning.The students learn from the system feedback and improve (player feedback).The players' feedbacks will influence the learning contents (maintain the current content, move to a more advance topic or go deeper into the same topic) and this will influence learning objectives and player goals.It was observed that game designs studied within the scope of the present research mostly used system feedback while ignoring player feedback.This is an indicator of failure of designs in terms of outcomes and feedback.
According to the findings, game designs developed by pre-service teachers were partially or completely satisfactory in terms of "Conflict (Competition, challenge, opposition)".Defining the game style in game design is very important in achieving the objectives of playing an educational game and forming the expected behaviours (Amory, 2001;Amory, Naicker, Vincent, & Adams, 1999;Ju & Wagner, 1997;Moreno Ger, Burgos, Martínez-Ortiz, Sierra, & Fernández-Manjón, 2008).The style that requires the cooperation between players to achieve the goals and the game style that requires competition between players with a winner and looser at the end are expected to be different.The findings of the present research show that pre-service teachers mostly preferred the competitionbased games in their designs and they successfully adapted competition element in their games.
Another finding of the present research was that, game designs developed by pre-service teachers were unsatisfactory in terms of "interaction".This result in failure in providing an intense interaction environment required for an effective learning environment, and therefore failure in achieving the learning objectives of the designed games (Pivec & Kearney, 2007).Additionally, pre-service teachers' failure in reflecting the interaction factor on their educational game designs can prevent their designs to be accepted as an educational game (Denis & Jouvelot, 2005).
According to the findings of the present research, games designed by pre-service teachers were partially or completely satisfactory in terms of "story".Pre-service teachers commonly stated that choosing contemporary stories and characters and effective environment were important and additionally there shouldn't be disconnections within story of the game.The more realistic the stories of the games are, the more connected can be the players with the game.In games, players can see themselves as heroes taking active parts in tasks (Song & Zhang, 2008).This is also considered as one of the most important elements providing players with motivation (Dickey, 2005(Dickey, , 2006;;Fisch, 2005;Waraich, 2004).Dickey (2006), stated that story context that stimulates the concepts of competition, imagination and curiosity increases internal motivation and participation related strategies can also increase motivation by covering options of role-play, story jumps, challenges and interaction.Despite the games designed within the scope of the present research lacked motivation due to unsatisfactory goal setting and feedbacks, partial success resulting from motivation in game designs can be associated with the satisfactory story setting.
The final finding of the present research was that game designs developed by pre-service teachers were also unsatisfactory in terms of "representation".This finding indicates that targeted educational situations weren't represented within the game.Game factor in pre-service teachers' game designs got ahead of educational objectives.However, the balance between educational objectives and game play should be well adjusted in educational game design (Kiili, 2005).Overall evaluation of all game design elements implies that pre-service teachers failed at achieving planned efficacy from educational games in their mathematical game designs.
Following suggestions were developed in accordance with the findings presented above: (1) Pre-service teachers should be provided with theoretical and practical knowledge on game design theories; (2) Pre-service teachers should be provided with opportunities to develop game designs in other teaching courses included in undergraduate classroom teaching program (Language Teaching, Social Sciences Teaching, Science Teaching); (3) Pre-service teachers should be provided with opportunities to apply their game designs on the appropriate audience in teaching practice courses.

4. 1
Introducing the game: This step involves the introduction of the game rules to the other students in the class.Researchers didn't interrupt at any point, but just observed.4.2 Application of the game: This step involves the playing of the game in the classroom environment.Researchers also didn't interrupt at any point, but just observed during this step. ).

Table 1 .
Evaluation of Game Designs in Terms of "Game Rules"

Table 2 .
Evaluation of Game Designs in Terms of "Goals and Objectives"

Table 3 .
Evaluation of Game Designs in Terms of "Outcomes and Feedback"

Table 4 .
Evaluation of Game Designs in Terms of "Conflict"

Table 5 .
Evaluation of Game Designs in Terms of "Interaction"

Table 6 .
Evaluation of Game Designs in Terms of "Story"

Table 7 .
Evaluation of Game Designs in Terms of "Representation"