Data Summary: DINO 101 Quantitative Study Fall 2013 Term Data

    A quantitative look at what happened in DINO 101 over the Fall 2013 semester.

    By Adam McCaffrey, Graduate Student, Educational Psychology; Lia Daniels, Associate Professor, Educational Psychology on June 11, 2014

    Description of the project: A study directed by Dr. Lia Daniels (University of Alberta) looked at applying motivational theories to understanding the levels and domains of engagement within the DINO 101 massive open online course (MOOC). This data collection was a preliminary look at students who completed the course and is the first step in what is a larger project that involves a longitudinal data collection in Winter 2014.

    Funding: This research was supported by Digital Learning Pilots: Research and Development at the University of Alberta.  

    Procedure: A total of 1,037 students were sampled through a voluntary link on the main page of the DINO101 website. The link was available at the last week of the course in early December until the official end of the course at the end of December. From this link students were connected to the SurveyMonkey© website, at which time they could complete a battery of questionnaires aimed at understanding their emotional and motivational experiences while learning within the MOOC environment.

    Data summary: The remainder of the document is intended as feedback to our course providers and participants. Thank you so much for your willingness to complete this survey. We have included information that we hope you find interesting and informative, as outlined below.

    Contents

    Section A: Description Participants

    Categories of participation

    Section B: Engagement

    Cognitive

    Behavioural

    Emotional

    Social

    Engagement x completion

    Differences: UofA vs DINO101 students

    Section C: Effort and Expectations

    Relationships with engagement

    Differences: UofA vs DINO101 students

    Section D: Final Thoughts





    Section A: Description of Participants

    Participants were 1037 students (50% female 50% male). This represents about 5% of the total number of students who were registered in DINO101 at any given time. Our sample also consisted of primarily DINO101 students (96.9%) who were not affiliated with the University of Alberta for credit PALEO 200/201 class. As displayed through Figure 1, the majority of participants were 25 to 34 years of age with 17% from 18 to 24 – the typical age range of undergraduate students.

    Figure 1. Age of Sample



    Figure 1. Age of Sample

    We also invited participants to indicate their highest level of education and to identify their level of current employment (Figures 2 and 3). Most participants were either working part-time or full-time and the majority had either completed an undergraduate or masters degree.

    Figure 2. Highest Level of Education



    Figure 2. Highest Level of Education

    Figure 3. Employment Status



    Figure 3. Employment Status

    Categories of Participation



    Previous literature describes different clusters of engagement among students in Massive Open Online Courses. The categories of participation we looked at exploring have been labeled Completers, auditors, dis-engagers and samplers (Kizilcec, Piech, & Schneider, 2013).



    – Completers: Those who completed the course fully either through their own timeline or the timeline structured through the course. In this study we broke this group into those who completed as to the class timeline and those who completed as to their own timeline.



    – Auditors: Those who completed the course but only did sections that they were interested in and did not engage in tests, assignments or grading.



    – Dis-engagers: Those who started the course with a level of engagement but this engagement lowered till these students eventually disengaged from the class either completely or to very little commitment.



    – Samplers: Those who simply joined the course to sample a few parts that they were interested in.

    As displayed in Figure 4 the majority of participants were “completers” as determined by their own setting their own timeline.

    Figure 4. Categories of Participation



    Figure 4. Categories of Participation

    Section B: Engagement



    The psychological literature defines engagement as a multi-dimensional concept describing how students actively commit to an action and participate in a process (Fredricks, Blumenfeld, & Paris, 2004). It is seen as being malleable and responsive to the learning environment – in other words course design and instruction can enhance or hinder engagement. Engagement is seen as an outcome of motivation and is a necessary for persistence in learning. Four of the most common dimensions of engagement are cognitive, behavioural, emotional and social engagement – all of which we considered for participants in DINO101.

    Cognitive Engagement: Cognitive engagement represents the thoughts and patterns of thinking that a student engages in that are necessary for learning. An example of this type of engagement is thinking about their work and answers, then mentally processing and working through problems to gain a deeper understanding.

    Looking at the distribution as based on the average inter-item rating, we found that cognitive engagement had a mean of 5.32, which positioned between “Neutral” and “Strongly Agree” that they engaged in this type of engagement (Figure 5).

    Figure 5. Inter-Item Mean for Cognitive Engagement



    Figure 5. Inter-Item Mean for Cognitive Engagement

    Based on our data the students in DINO scored a total mean of 26.6 (SD= 5.6) out of a maximum score of 35 (Figure 6). This suggests that they generally agreed that they were cognitively engaged in the course with some skewness towards agreeing strongly towards being cognitively engaged and very few saying that they disagreed with being cognitively engaged.

    Figure 6. Total Distribution of Cognitive Engagement



    Figure 6. Total Distribution of Cognitive Engagement

    Behavioural Engagement: Behavioural engagement represents a type of engagement that specifically looks at the observation and physical attributes of engagement that are needed to learn. An example of a behavioural engagement can be that the student was able to physically sit and focus their attention to the computer while online learning.

    Looking at the distribution as based on the average inter-item rating, we found that behavioural engagement had a mean of 5.23, which positioned between “Neutral” and “Strongly Agree” that they engaged in this type of engagement (Figure 7).

    Figure 7. Inter-Item Mean for Behavioural Engagement



    Figure 7. Inter-Item Mean for Behavioural Engagement

    Based on our data the students in DINO with a mean score of 31.4 (SD= 5.4) out of a maximum score of 42 (Figure 8). This suggests that they generally agreed that they were behavioural engaged in the course with some skewness towards agreeing strongly towards being behavioural engaged and very few saying that they disagreed with being behavioural engaged.

    Figure 8. Total Distribution of Behavioural Engagement



    Figure 8. Total Distribution of Behavioural Engagement

    Emotional Engagement: Emotional engagement represents a type of engagement in which a student feels an emotional response to the learning that helps them to maintain or foster a sense of interest and persistence in learning. An example is whether a student feels they were able to maintain a level of excitement about the material, which could act to motivate them to persist in the learning process.

    Looking at the distribution as based on the average inter-item rating, we found that emotional engagement had a mean of 6.2, which positioned closest to “Strongly Agree” that they engaged in this type of engagement (Figure 9).

    Figure 9. Inter-Item Mean for Emotional Engagement



    Figure 9. Inter-Item Mean for Emotional Engagement

    Based on our data the students in DINO with a mean score of 31.0 (SD= 4.8), out of a maximum score of 42 (Figure 10). This suggests that students leaned towards strongly agreeing that they were emotional engaged in the material. There was a large skewness towards strongly agreeing that the course maintained an emotional environment that was able affect them in maintaining a positive emotional experience.

    Figure 10. Total Distribution of Emotional Engagement



    Figure 10. Total Distribution of Emotional Engagement

    Social Engagement: Social engagement represents a type of engagement in which the student feels a sense of belonging and social connection that helps to motivate a student to persist and maintain interest in a course. An example would be of a student being able to connect with other students in the online environment by sharing and helping others.

    Looking at the distribution as based on the average inter-item rating, we found that social engagement had a mean of 3.6, which positioned between “Neutral” and “Strongly Disagree” that they engaged in this type of engagement (Figure 11).

    Figure 11. Inter-Item Mean for Social Engagement



    Figure 11. Inter-Item Mean for Social Engagement

    Based on our data the students in DINO with a mean score of 14.4 (SD= 5.0) out of a maximum score of 28 (Figure 12). This suggests that students generally were neutral on whether they felt they were able to socially engage with other students in the course. There was a slight skewness of students that disagreed that they were able to socially engage with other students.

    Figure 12. Total Distribution of Social Engagement



    Figure 12. Total Distribution of Social Engagement

    Levels of Engagement by Participation Category



    Overall, our participants were asked to rate themselves on how true the statements were of themselves during the course. For each of the different types of engagement we divided the total score by the number of questions used for that specific engagement type. This calculation created an inter-item score that can compare each engagement type on a 7-point scale.

    Looking at Figure 13 we see that in the areas of cognitive and behavioural engagement, participants rated themselves as being generally agreeing to being engaged in these behaviours. Emotional engagement showed the highest agreement amongst participants and suggests that the DINO online environment appears to be a safe and exciting place for the majority of participants. The lowest component of engagement was the social domain showing a slight disagreement that they were able to socially engage with the material and other students. This suggests that while the online environment appears to be emotionally, cognitively and behavioural engaging, students did not perceive the course as socially engaging.

    Figure 13. Engagement Dimensions by Participation Category



    Figure 13. Engagement Dimensions by Participation Category

    PALEO200/201 Versus DINO101 Students



    Our sample consisted of both DINO101 students who were from around the world and entered the course through the coursera platform and the PALEO200/201 students who were a part of the same online learning environment but were completing the course for actual University of Alberta credits and had an on campus teaching assistant. Figure 14 compares the levels of engagement across the two different course categories and reveals that there is little difference between the groups in terms of engagement levels. The DINO101 students do appear to have slightly higher levels of engagement and this might be due to the different intentions for taking the class (for interest versus course credit).

    Figure 14. Engagement Dimensions by Course Category



    Figure 14. Engagement Dimensions by Course Category

    Section C: Effort and Expectations



    Effort and expectations are two foundational constructions in the area of achievement motivation (Eccles, Wigfield, & Schiefele, 1998; Weiner, 1986). Students perform better when they have high expectations and high expectations are often positively correlated with greater effort expenditure. Well-invested effort can be the key to successful learning. We asked students about how much effort they invested in this course and a variety of their expectations. Means scores on the individual items are reported in Figure 15. Effort appeared to be rated slightly above neutral suggesting that students put some effort into the course but were not overburdened. Expectations and satisfaction in the course were for the most part rated highly, with the exception that they did not expect that they would be learning through interacting. We found this surprising given the “massive” and “open” intention of MOOCs. Although this is truer for cMOOCs than xMOOCs, we nonetheless were surprised that very little learning was expected to come from the other 20,000+ learners.

    Figure 15. Effort and Expectations Bar Graph



    Figure 15. Effort and Expectations Bar Graph

    Correlational Analysis of Engagement to Effort, Expectations and Satisfaction

    Engagement in the previous section is an important component to determining students’ effort, satisfaction and expectations for the course. Below are the correlations to assess the relations between engagement, effort, satisfactions and expectations. Although all correlations were statistically significant (as would be expected), the role of emotional engagement is particularly important to point out since it had the strongest correlation with both expectations and satisfaction. Thus, courses that emotionally engage students may be best poised to have students are satisfied. Both cognitive and behavioural engagement appears to be most strongly related to effort. Thus, to create a course in which students invest a lot of effort, they need opportunities to be cognitively and behaviourally engaged in the materials.

    Table 1. Correlations Between Engagement, Effort & Expectations



    Pearson Correlations, *p < .001



    Table 1. Correlations Between Engagement, Effort & Expectations

    T-tests Between DINO101 and PALEO200/201



    To analyze whether there was a difference between the DINO101 and the PALEO200/201 participants were ran a few independent sample t-tests to compare mean differences. Table 2 shows that there was a statistical difference between the DINO101 group to the PALEO200/201 in terms of the expectations they had for the course. In this case the DINO101 students had higher expectations for learning in the course than the PALEO200/201 class. In terms of effort and satisfaction there was no statisticl difference.

    Table 2. Independent Samples t-test for Expectations, Effort & Course Satisfaction by Course category



    Table 2. Independent Samples t-test for Expectations, Effort & Course Satisfaction by Course category

    Section C: Final Thoughts



    – DINO101 was a highly engaging course from the student perspective – however, social engagement was both the lowest rated and seemed the least relevant in terms of relationships with effort or satisfaction.



    – At least within DINO101 students did not expect social connections to be a source of their learning. Given the massive and open nature of MOOCs this is somewhat surprising and contrary to social psychological theories that reinforce the importance of this type of engagement. However based on the overwhelming positive response to DINO101, we can only imagine that tapping into social engagement would make the course even more appealing to students. The finding that “samplers” actually had higher levels of social engagement than other categories suggests that perhaps this is part of the key to retaining learners.



    – The preliminary results presented herein largely reflect the responses of students who completed DINO101 – or at least visited the course again towards its conclusion. Thus, information from students who left the course part way through is missing and this valuable perspective could help inform course design to help retain even more learners.