Multi-Modal Video Learning Platform

The Problem

Passive video consumption dominates online learning, yet research on multimedia learning principles (Mayer, 2009) shows that engagement and retention drop sharply when learners watch without interacting. Most video platforms treat learners as viewers rather than participants — there is no mechanism to adapt pacing, check understanding, or provide alternative explanations when a concept does not land.

Approach

This platform layers interactive affordances on top of video content to create a closed-loop learning experience. The system analyzes video segments to identify conceptual boundaries, then inserts lightweight knowledge checks, branching explanations, and supplementary visual overlays at precisely the right moments.

Key capabilities include:

• Automatic segmentation of video content into conceptual units using transcript analysis
• Embedded assessments — short, contextual questions that appear at segment boundaries and gate progression
• Adaptive branching — when a learner struggles with a concept, the system surfaces alternative explanation clips or supplementary diagrams
• Multi-modal reinforcement — synchronized visual annotations, highlighted transcripts, and summary cards that engage multiple channels simultaneously

My Role

I led the design and implementation of the adaptive engine, the video segmentation pipeline, and the interactive overlay system. I also designed the learner model that drives branching decisions based on real-time performance signals.

Results & Impact

• Learners demonstrated higher post-assessment scores compared to the same content delivered as passive video
• Average completion rates increased as adaptive branching kept learners in their zone of proximal development
• The overlay system works with any standard MP4 source, requiring no special video production tooling

Technical Details

• Video Pipeline: FFmpeg-based segmentation with transcript-aligned boundary detection
• Adaptive Engine: Bayesian knowledge-tracing model that updates learner state after each interaction
• Frontend: Lightweight custom video player with Canvas 2D overlay for annotations and interactive elements
• Assessment Authoring: LLM-assisted question generation from transcript segments, with human review workflow
• Analytics: Event-stream architecture capturing interaction-level data for learning analytics dashboards