Data that Drive Better Learning

Oct 7, 2020 11 min read Learning Analytics, Educational Data

Educational data collected from learning management systems, digital courseware, and other technology provide ample opportunity to explore associations between academic success and student performance and behavior. But the informativeness and actionability of this data, particularly as it pertains to improving teaching and learning, is widely regarded as poor.1

Data is often collected and interpreted with little consideration of an instructor’s pedagogical intentions, absent valid measures of learning impact, and without consulting available learning research. As a result, many learning analytics ‘solutions’ may unintentionally support inferior learning experiences.

Consider the following:

How valuable are accurate predictions of course success when even students who pass a course learn very little? How informative is data on student performance without clear links between course assessments and intended learning outcomes? *How useful are early indicators of student risk when subsequent interventions conflict with research on how to provide effective feedback?

Fortunately, we can do better!

Learning analytics will have the greatest positive impact when decisions about data collection and analyses are guided by research in the educational sciences and learning outcomes are carefully measured to assess impact. Thus any learning analytics project should start by answering the following series of questions:

*What information about students’ performance and behaviors can be tied back to learning research and are associated with actions known to improve student learning, motivation, and success?

*What types of data and recordable student activities can surface this information so that faculty, advisors, instructional designers, and others are empowered to make evidence-informed changes that reliably improve learner outcomes?

*How can we measure the quality of a learning experience and evaluate the impact of data-informed changes to continually improve the learning experiences we design for students?

Answering these questions will point us to data that can drive better learning.

So, what educational data are most valuable for instructors and learning designers wanting to improve student outcomes?

Below you will find quick summaries of four data themes that emerge when we seek answers to the questions outlined above. Within each section, I share the research-based rationale for collecting the data described while also hinting at productive actions you can take in response to data findings. Finally, I provide practical recommendations on how you can start collecting meaningful and actionable educational data in your courses today!

Know Thy Students

Key Questions

*Are students prepared to succeed in your course?

*How motivated & confident are students in their ability to do well?

*Do incoming students use effective studying and self-monitoring strategies?

Research Says

Students entering your course have different levels of subject matter expertise, mindsets, and motivations. Some students will be confident in their ability to perform well in your class, while others may doubt their ability to succeed; some students will have a strong background in course material, while others have misconceptions that will make learning more challenging. These cognitive and non-cognitive attributes are some of the most powerful known factors impacting student academic success.2 Students’ prior knowledge, attitudes, and beliefs are the key foundation upon which all future learning builds.3

Taking Data-Informed Action

Understanding your students’ backgrounds and beliefs enables you to design a learning experience that effectively engages students’ prior knowledge, addresses unproductive mindsets, and fills critical skill gaps.4,5 If you find students have poor study strategies or lack confidence in their ability to succeed, you can devote time early in a semester to activities that address these obstacles.6 Insight into student mindsets and learning beliefs can be used to tailor feedback that maximizes learner receptivity and increases student feelings of self-efficacy.7 Awareness of common learner misconceptions and errors can help you sharpen your instructional focus and employ methods known to effectively redress erroneous beliefs.8 And data on student progression across a series of courses may reveal the need to reevaluate current course sequencing and/or content coverage if students are not leaving and entering courses adequately prepared.

Example of course pathways analysis showing the courses students take most frequently after a given course—this type of analysis ensures course content is aligned with student interests and adequately prepares them for later courses.

Getting Started

Administer diagnostic exams and concept inventories at the beginning of a course Give students learning skills/strategies questionnaires early in a course Examine student course pathways through your course and subsequent courses Survey student mindsets and non-cognitive attributes

Illuminate a Clear Learning Path

Key Questions

How successfully are students meeting course learning objectives? Where does course difficulty and effort increase in your course? *Are course learning goals meaningful and relevant to students?

Research Says

Clear, meaningful, and challenging goals are key to an effective learning experience.9 Students will be optimally motivated and engaged when course objectives are clearly defined, there exists strong learner commitment to achieving learning goals, and students are confident in their ability to succeed.10,11 Students benefit from frequent updates on their progress combined with specific recommendations on how to best direct their efforts going forward.13 Given the poor metacognitive skills of many students, explicit guidance to aid students in planning, monitoring, and adjusting their learning efforts also significantly improves learner achievement.

Taking Data-Informed Action

Carefully align course learning objectives, assessments, and learning activities to create strong cohesion in your course.14 Provide students with regular updates on their progress toward achieving course learning goals— including highlighting things done well and areas of growth. Share expectations with students regarding the amount of time needed to complete upcoming assignments to help students plan and regulate their studying activities.15 Send anticipatory encouragements and study tips prior to jumps in course difficulty to help students remain confident in their ability to succeed. Finally, use information about student interests and goals to motivate your course learning objectives and increase learner engagement.16

Example of a visualization showing a substantial drop in student homework performance—an excellent opportunity to provide anticipatory encouragements and/or devote additional instructional time.

Getting Started:

Explore course student activity and performance trends Monitor student learning performance on aligned assessments/activities Collect and analyze the time students spend completing assignments and activities Administer questionnaires on student motivations, interests, and goals

Support Student Growth

Key Questions

Are students regularly required to demonstrate and evaluate their understanding? Do students have the means and motivation to apply received feedback? Are students encouraged to reflect on the effectiveness of their learning efforts?

Research Says

Students need to practice applying their knowledge along with frequent and timely feedback.17 Designing instruction so that students are required to regularly demonstrate their understanding allows you to monitor student learning progress, identify gaps in comprehension, and improve student retention.18 Frequent small and multi-stage assessments enable the provision of meaningful feedback by giving students the motivation, opportunity, and means to use the feedback they receive.13 And students learn most deeply when given the opportunity to reflect on their current state of knowledge and regularly evaluate the effectiveness of their learning efforts.3

Taking Data-Informed Action

Assign early and frequent low-stakes assessments– such as weekly quizzes or small scope tasks – to track learner progress and identify patterns of student error. Performance on these assignments will allow students to monitor their learning progress and give you the opportunity to provide specific feedback to help learners adjust their studying efforts, particularly important for less prepared students.19 Require that students provide evidence of incorporating received feedback, using multi-part assignments and submission of rough drafts, to encourage student reception and application of the feedback you provide.20 Finally, collect data on student study strategies and learning reflections, using brief journal assignments and/or exam wrappers, to identify areas of confusion and struggle while at the same time encouraging learner self-monitoring.22

Example of a heatmap showing student posting activity in a course—this type of analysis can be used to help identify concerning patterns in student behavior and encourage students to adopt more successful learning strategies (e.g., spacing studying rather than cramming).

Getting Started:

Explore student activity and performance trends in your course for concerning patterns Monitor student learning outcome performance on aligned assessments/activities Collect and analyze the time students spend completing assignments and activities Administer questionnaires on student motivations, interests, and goals

Understand Your Impact

Key Questions

How much do students grow between the beginning and end of your course? What impact do curricular changes and student characteristics have on student outcomes? How do students perceive the organization, pace, and climate of your course?

Research Says

Effective teaching involves a continuous cycle of monitoring student achievement, assessing the effectiveness of instructional design changes, and adjusting course design.15 Critical to this process is a clear understanding of your course’s impact on student outcomes. Valid and reliable measures of student learning are essential for accurately evaluating and improving your teaching efforts.22 Eliciting regular feedback from students on your instructional methods and course climate can also provide valuable insights to guide further improvements.15

Taking Data-Informed Action

An important first step to understanding your impact is measuring student learning gains on key course concepts and skills.23 Systematic efforts to evaluate the effectiveness of interventions and curricular changes are also critical for driving incremental course improvements. Examining associations between course outcomes and student characteristics (e.g., gender, socio-economic status, first-generation status) can validate assumptions about incoming students and identify problematic patterns of differential impact.24 Finally, using feedback collected from students throughout the semester on both teaching (e.g., clarity, pacing, & organization) and course climate can help you better understand course strengths and areas for improvement.

Example of a visualization showing student performance changes between a pre and posttest on key learning objectives in a course—this type of analysis can help you evaluate whether or not instructional interventions are making a positive impact and assess student learning gains.

###Data to get started:

Collect pre and posttest data on key course outcomes Administer surveys to gather feedback on your teaching effectiveness & course climate Measure relevant learning and behavioral data to assess the impact of course changes Probe inequity in your courses by looking for associations between student performance and demographic/background characteristics

References

  1. Ferguson, R., & Clow, D. (2017). Where is the evidence? A call to action for learning analytics. In Proceedings of the Seventh International Learning Analytics & Knowledge Conference on - LAK ’17 (pp. 56–65).

  2. Ambrose, S. A., & Lovett, M. C. (2014) Prior Knowledge is More Than Content: Skills and Beliefs Also Impact Learning. In Applying the Science of Learning in Education. APA.

  3. Burnette, J. L., O’Boyle, E. H., VanEpps, E. M., Pollack, J. M., & Finkel, E. J. (2013). Mind-sets matter: A meta-analytic review of implicit theories and self-regulation. Psychological Bulletin, 139(3), 655–701.

  4. Sawyer, K. (Ed.). (2014). The Cambridge handbook of the learning sciences (2nd ed.). New York: Cambridge University Press.

  5. Credé, M., & Kuncel, N.R. (2008). Study habits, study skills, and study attitudes: The third pillar supporting collegiate academic performance. Perspectives on Psychological Science, 3, 415-453.

  6. Hattie, J., Biggs, J., & Purdie, N. (1996). Effects of learning skills interventions on student learning: A meta-analysis. Review of Educational Research, 66, 99–136.

  7. Lazowski, R. A., & Hulleman, C. S. (2015). Motivation interventions in education: A meta-analytic review. Review of Educational Research, 39, 1–39.

  8. Kowalski, P., & Taylor, A. (2009). The effect of refuting misconceptions in the introductory psychology classroom. Teaching of Psychology, 36(3), 153-159.

  9. Locke, E. A., & Latham, G. P. (2006). New Directions in Goal-Setting Theory. Current Directions in Psychological Science, 15(5), 265–268.

  10. Hattie, J., & Timperley, H. (2007). The Power of Feedback. Review of Educational Research, 77(1), 81–112.

  11. Hattie, J., & Yates, G. C. (2013). Visible learning and the science of how we learn. Routledge.

  12. Forsythe, A., & Johnson, S. (2017). Thanks, but no-thanks for the feedback. Assessment and Evaluation in Higher Education, 42(6), 850–859.

  13. Shute, V. J. (2008). Focus on formative feedback. Review of educational research, 78(1), 153-189.

  14. Biggs, J. (2014). Constructive alignment in university teaching. HERDSA Review of higher education, 1(1), 5-22.

  15. Ambrose, S., Bridges, M., Lovett, M., DiPietro, M., & Norman, M. (2010). How Learning Works: 7 Research-Based Principles for Smart Teaching. San Francisco, CA, USA: John Wiley & Sons, Inc. 

  16. Järvelä, S., & Renninger, K. (2014). Designing for learning: Interest, motivation, and engagement. In Keith Sawyer (ed), The Cambridge Handbook of the Learning Sciences. Cambridge University Press. p. 668.

  17. Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological bulletin, 119(2), 254.

  18. Roediger, H. L., Putnam, A. L., & Smith, M. A. (2011). Ten Benefits of Testing and Their Applications to Educational Practice. Psychology of Learning and Motivation, 55, 1–36.

  19. Harrison, C. J., Könings, K. D., Schuwirth, L., Wass, V., & van der Vleuten, C. (2015). Barriers to the uptake and use of feedback in the context of summative assessment. Advances in Health Sciences Education: Theory and Practice, 20(1), 229–245.

  20. Carless, D., & Boud, D. (2018). The development of student feedback literacy: enabling uptake of feedback. Assessment & Evaluation in Higher Education, 2938(May), 1–11. http://doi.org/10.1080/02602938.2018.1463354

  21. McInerney, V., McInerney, D. M., & Marsh, H. W. (1997). Effects of metacognitive strategy training within a cooperative group learning context on computer achievement and anxiety: an aptitude- treatment interaction study. Journal of Educational Psychology, 89, 686-695. doi:10.1037/0022-0663.89.4.686

  22. Gelman, A., & Loken, E. (2012). Ethics and Statistics: Statisticians: When We Teach, We Don’t Practice What We Preach. Chance, 25(1), 47–48. Pentecost, T. C., & Barbera, J. (2013). Measuring learning gains in chemical education: A comparison of two methods. Journal of Chemical Education, 90(7), 839–845.

  23. Matz, R. L., Koester, B. P., Fiorini, S., Grom, G., Shepard, L., Stangor, C. G., … McKay, T. A. (2017). Patterns of Gendered Performance Differences in Large Introductory Courses at Five Research Universities. AERA Open.

Learning Science Educational Data Learning Analytics
Jay Lynch
Jay Lynch
Educational Data Scientist

My interests include learning analytics, data visualization, and supporting greater educational equity.

Related