Photo by Chris Liverani on UnsplashEvaluation
Table of Contents
Evaluation and implementation are essential pieces of the puzzle for engaging learning experiences. Alignment of both aspects will allow for the intentional use of selected materials.
Technology Impacts
There are multiple integration frameworks available to review media, technology, and other educational materials within the online environment. Each of these frameworks provides a rubric or analytical lens to view specific aspects of implementation or integration. Concerning evaluation, it is a process to perform before, during, and after. It is also helpful to involve learners in the process, if at all possible. Some directions on the evaluation of technology within the online learning environment are available through the Rubric for eLearning Tool Evaluation by Anstey & Watson (2018) and The Technology Integration Matrix by the Florida Center for Instructional Technology at the University of South Florida.
The rubric developed by Anstey & Watson (2018) examines aspects of functionality, accessibility, technical, mobile design, privacy, data protection and rights, social presence, teaching presence, and cognitive presence. While in contrast, The Technology Integration Matrix evaluates the level of engagement and implementation of the technology’s use. Taking the topic of evaluation further, we should also consider Mayer’s twelve principles of multimedia (DeBell, 2019) when including multimedia to online learning.
Consider as many evaluation points or your preferred rubrics for assessment while also requesting feedback from your stakeholders. For example, let’s consider adding a video player on a content page within an online course. It is a straightforward piece of content to embed, and many video hosting websites provide an easy-to-copy and paste embed code. But simply evaluating for accessibility can provide usability factors.
- The video should not auto-play
- The video controls should always be present
- Accurate captions are available for the video
- The video player is keyboard accessible
- Does the video player offer audio description as an audio option?
- Can the user view the video on full-screen?
- The video does not provide rapid flashing
The above is one set of factors to consider for one type of media, video, for implementation into a lesson page. The difference is some learners accessing the video or all learners accessing the video.
Example Expanded
Let’s take the video example further by using YouTube. YouTube is one of the most popular video hosting services available, and users can find content on virtually any topic. The video player for YouTube is keyboard accessible. However, not all videos provide accurate captioning, and it can also be challenging to contact the content creator to request or deliver accurate captions for upload. Many educational institutions will partner with a captioning provider to ensure accurate captions and have a video player service integrated with the Learning Management System. It is possible to embed a YouTube video within a content page with accurate captions utilizing these two services. If your institution does not offer this capability, Amara may be a suitable option. Amara is a free platform to caption and translate video captions for YouTube videos. Each video on Amara has a unique link to provide on the content page. It may not be possible to directly embed the video within a content page with script sanitizing. However, linking out to the accurately captioned video opens access to all, and there are many community volunteers to provide translations to other languages.
While there is not an example lesson here, many courses utilize video to supplement and enhance course content to provide an alternative form of access to the information. Some videos give the entire lecture for a course module. The learning objectives that apply may be recalling, listing, describing, explaining, or other Bloom’s taxonomy verbs to have learners provide a measurable learning outcome from viewing the video.
For an example video, we will provide the YouTube Algorithms: How to Avoid the Rabbit Hole | Above the Noise video with ISTE learning outcome 4.a “Students: know and use a deliberate design process for generating ideas, testing theories, creating innovative artifacts or solving authentic problems.”
We can see the video player is keyboard accessible, the video does not autoplay, there are accurate captions, and users can easily make the video play in full-screen. After we have established those criteria are met. We can further review the media against the overall content of the lesson, ensure the video is embedded consistent with local design standards, and align both the video and content to the overall learning objectives for the course, module, and lesson.
Personal Analysis
Evaluating YouTube content through the Rubric for eLearning Tool Evaluation by Anstey & Watson (2018) results in the following:
| Category | Criteria | Rating | Notes |
|---|---|---|---|
| Functionality | Scale | Works Well | Instructors can share YouTube content to an entire class through LMS content such as pages, announcements, assignments, and personal messaging. |
| Functionality | Ease of Use | Works Well | YouTube provides a user-friendly interface that is easy for both learners and instructors to become familiar with using. In addition, minimal customization is available for the player. |
| Functionality | Tech Support / Help Availability | Works Well | Technical documentation is publically available through YouTube Help in a variety of languages. In addition, most local Information Technology support personnel will assist with standard solutions related to video posting settings. |
| Functionality | Hypermediality | Minor Concerns | While the video platform does support video hotspots and annotations produced by an author, the interactivity with the content is limited and not available for branching. |
| Accessibility | Accessibility Standards | Minor Concerns | YouTube provides the ability to upload caption and subtitle files and will automatically generate captions from the video source language. However, automated captions are not as accurate as human transcription. YouTube also does not provide the ability to add an audio description soundtrack. |
| Accessibility | User-focused participation | Not Applicable | The original content authoring provides the ability to provide opportunities for participation in learning. |
| Accessibility | Required Equipment | Works Well | YouTube does not require additional equipment to function beyond a current web browser, speakers, microphone, and internet connection. |
| Accessibility | Cost of Use | Minor Concerns | Advertising and requests to try out the free trial of YouTube Premium can be intrusive and require repeated dismissal. Some aspects of the platform do require subscription services for feature films and television episodes. |
| Technical | Integration / Embedding within a Learning Management System (LMS) | Works Well | Most Learning Management Systems support the direct embed of YouTube video content via an embed code. I recommend using the embed code method with no tracking, see the support article on Embedding videos using YouTube's Privacy Enhanced Mode |
| Technical | Desktop / Laptop Operating Systems | Works Well | Users can effectively utilize YouTube with any standard, up-to-date browser on any operating system. |
| Technical | Browser | Works Well | Users can effectively utilize YouTube with any standard, up-to-date browser. |
| Technical | Additional Downloads | Works Well | Users do not need to download additional software or browser extensions. However, many browser extensions are available to customize the YouTube viewing experience. |
| Mobile Design | Access | Works Well | Users can access YouTube via a mobile web browser, mobile app for iOS and Andriod, and various other platforms like Amazon Fire TV products, Roku, XBOX, and other systems. Thus, their platform works well on different screen sizes, including smaller screens. |
| Mobile Design | Functionality | Works Well | The user experience is nearly identical on mobile devices as it is on regular laptop and desktop computers. |
| Mobile Design | Offline Access | Serious Concerns | Offline access to content is not available for free accounts. |
| Privacy, Data Protection, and Rights | Sign Up / Sign In | Serious Concerns | Users do not need to sign up for an account to access video content. However, requiring a student to sign up for an account does collect personally identifiable information to YouTube as a third-party. |
| Privacy, Data Protection, and Rights | Data Privacy and Ownership | Not Applicable | If considering adopting a paid account for YouTube, run the terms and conditions past your local Legal Counsel. The amount of ownership is a fuzzy area when uploading content to YouTube as they will take down content. Users can place videos as private, but if there are segments of copyrighted material within the video, YouTube may not allow for that video to receive a shareable link. |
| Privacy, Data Protection, and Rights | Archiving, Saving, and Exporting Data | Works Well | Users can upload content in a variety of file formats. However, the user can only download or "back up" their content in the originally uploaded format. |
| Social Presence | Collaboration | Works Well | YouTube supports live streaming with interaction in the form of a text chat for comments and limited reactions. The platform is mainly available to broadcast and not have others interactively join a session. Live streams do support the integration of Zoom and other streaming providers to publish an interactive session live and make the video available as an asynchronous tool. |
| Social Presence | User Accountability | Not Applicable | |
| Social Presence | Diffusion | Works Well | YouTube is widely known and popular. It is likely that most learners are familiar with the tool and have basic technical competence. |
| Teaching Presence | Facilitation | Works Well | YouTube works well at promoting teaching presence in an asynchronous environment. However, it is not applicable in the live classroom. |
| Teaching Presence | Customization | Works Well | Instructors can customize YouTube to fit the classroom context. I have had instructors provide YouTube videos to share feedback on my work. I have seen it used to provide teacher presence in announcements and for small groups. |
| Teaching Presence | Learning Analytics | Not Applicable | |
| Cognitive Presence | Enhancement of Cognitive Task(s) | Serious Concerns | YouTube videos replace an interaction or lecture within a traditional classroom-based environment. Depending on the content, little to no cognitive engagement may result from viewing a video. |
| Cognitive Presence | Higher Order Thinking | Works Well | Instructors can post videos that promote higher-order thinking, depending upon familiarity with video authoring tools. |
| Cognitive Presence | Metacognitive Engagement | Not Applicable |
Evaluating YouTube content through The Technology Integration Matrix by the Florida Center for Instructional Technology at the University of South Florida depends upon the degree to which the technology is integrated or implemented in the classroom. Simply providing a video on a page for a learner to view for information would result in an “active learning, entry level” where information is passively received. The extended descriptors from the matrix are as follows:
- Students. Students receive information from the teacher or from other sources. Students may be watching an instructional video on a website or using a computer program for drill and practice activities.
- Teacher. The teacher may be the only one actively using technology. This may include using presentation software to support delivery of a lecture. The teacher may also have the students complete drill and practice activities on computers to practice basic skills, such as typing.
- Setting. The setting is arranged for direct instruction and individual work. Any student access to technology resources is limited and highly regulated.
I agree, depending upon the content and classroom context. Students can use YouTube videos to share digital portfolio content, present their ideas, and promote twenty-first-century skills. The primary basis of evaluation is on the posting of YouTube videos within LMS content. The Technology Integration Matrix provides other levels of engagement and levels of adoption.
References
Above The Noise. (2019, June 19). YouTube algorithms: How to avoid the rabbit hole [Video]. YouTube.
Anstey, L. & Watson, G. (2018, September 10). A rubric for evaluating e-learning tools in higher education. EDUCAUSE Review.
DeBell, A. (2019, December 11). How to use Mayer’s twelve principles of multimedia, examples included. Water Bear Learning.
Florida Center for Instructional Technology. (n.d.). The technology integration matrix. University of South Florida.
Steven Kolberg
Senior Coordinator of Accessibility and LMS Administration
My research interests include instructional design, web accessibility, computer programming, and education.