From Data to Action: Bruin Learn Analytics in Your Course
This showcase demonstrates the tools and reports Bruin Learn provides to data on student engagement, participation, and achievement. Through Bruin Learn analytics, the presenters explain how to measure how students engage with course activities, identify at-risk students and assess the effectiveness of teaching strategies. The limitations of analytics and how your course design will impact your ability to gather meaningful data have also been discussed.
What to Expect in this Showcase
You will learn about the tools and reports Bruin Learn provides to data on student engagement, participation, and achievement. Bruin Learn analytics can:
- measure how students engage with course activities,
- identify at-risk students and
- assess the effectiveness of teaching strategies.
We will also discuss the limitations of analytics and how your course design will impact your ability to gather meaningful data.
Presenters
Jason and Mark worked together on the recent LMS Transformation project that moved over 40,000 course sites from CCLE/Moodle into Bruin Learn/Canvas.
Mark Kayser is an Instructional Designer with the UCLA Teaching and Learning Center. In his work he has designed online and blended courses with a special interest in STEM courses and the unique challenge these domains pose to online learners. For the last 15 years he has also taught Mathematics courses at the collegiate level.
Jason Schaffer is an LMS Analyst with UCLA Bruin Learn Center of Excellence (CoE). Jason has worked closely with numerous academic partners to integrate 3rd party LTI tools, in addition to feature options in Bruin learn. He has helped support Canvas at UCLA since the inception of the CoE.
Selected Q&A from the Showcase
| Question | Answer |
|---|---|
| Can you explain more about the 30-day limitation? It sounds like the data will only reflect what happens in 30 days? Relatedly, will Canvas make a change to this limitation? | The 30 days are a rolling window. The idea is for you to pay attention to the data throughout the course to make more real-time decisions based on real-time(ish) data. There are ways to download more data, but there are no plans that we know of for Canvas to change this limitation. The company does make frequent updates so we’ll see. |
| Can you clarify what weekly/online activities will be counted? | Files, PDF are not counted. Those with participation are. |
| How do page views work with files (is it any different)? I’ve just observed page views being pretty low in a class with lots of files for readings. (But maybe students are actually not viewing them!) | If the files are not a Canvas page, that won’t trigger in New Analytics. |
| In what ways could you miss a student not doing well who seems to be doing well as per the data? | Consider implementing low-stakes assessments in your course design. For example, if you’re only giving a traditional high-stakes midterm, then you won’t see student progress until the midterm scores. |
| How can faculty learn about these powerful formative assessment tools, like tracking student engagement? | Both the Teaching and Learning Center and the Bruin Learn COE offer consultation, showcases, workshops, and training: |
| Will New Analytics also include info from TA sites? | The New Analytics are specific to the course you’re in – to see a TA site’s New Analytics, you can find them in that particular site. |
| What about the site from a Group Set? (When you create groups, this creates sites within the main site.) | We will need to test this! |
| If you’re teaching an in-person course, how can you design your class site and teach it in a way to make use of New Analytics? | Consider your goals for using analytics. You’d likely want to know if students are accessing the resources – so, for instance, you might want to embed the readings into a page so that you get page view data. You could also have some low-stakes assessments (pre-lecture quizzes etc) in order to get some data. |
| How does it count things like Perusall? | New Analytics doesn’t cover applications originating outside Canvas, but many of these apps have their own data. For example, you can see Perusall’s data through the Perusall tab.
Each external third-party tool is going to be different in the data it provides. |
| Are Kaltura analytics available through New Analytics? | Those analytics are to be found separately. In “My Media,” by clicking on the graph next to one of your videos, you’ll be able to see some data.Treat the analytics as “clues” and not absolute proof! |
| For the Canvas analytics, because of the 30-day limitation, does it mean it requires lots of effort from the instructor to track the data? What best practices would you suggest if they want to collect and use the data? | Think of New Analytics less as a replacement for the info in the Gradebook and more like a temperature check. It is also not hard and fast evidence. New Analytics can help you reduce uncertainty about your course.
Your academic unit lead might have access to additional reports/data that are outside the course. |
| This is not about New Analytics, but do you have anything to say about the “time spent” value in the People tab? How would you interpret that info? | This data point can be misleading because students can leave their Bruin Learn browser tabs open and not performing any course activity. On the other hand, if the value is super low it may indicate your student hasn’t logged in to the course site. It also lets you see if there was any access at all. |
This Google Doc contains the comments, questions and answers collected during the showcase.