Temporal Alignment of Human Motion
Temporal alignment of three subjects kicking a ball.
These three sequences are recorded with different sensors (top row video, middle row motion capture and bottom row accelerometers).
People
- Feng Zhou (CMU)
- Fernando De la Torre (CMU)
Introduction
Temporal alignment of human motion has been a topic of recent interest due to its applications in animation, tele-rehabilitation and activity recognition among others. This paper presents canonical time warping (CTW)[1][2][3], an extension of dynamic time warping (DTW)[4] for temporally aligning multi-modal sequences from multiple subjects performing similar activities. CTW solves three major drawbacks of existing approaches based on DTW:
- CTW provides a feature weighting layer to adapt different modalities (e.g., video and motion capture data);
- CTW extends DTW by allowing a more flexible time warping as combination of monotonic functions;
- Unlike DTW that typically incurs in quadratic cost, CTW has linear complexity.
Code
Available at here.
Result
Temporal alignment of three multi-modal sequences by GTW.
The first two sequences are taken from CMU Motion Capture Database and Weizmann Action Database. The last sequence records the hand movement by an accelerometer. You can reproduce the same result using the function demoMix.m in the code.
Download the [Video 50MB].
Temporal alignment of two motion capture sequences by CTW.
The motion capture sequences are taken from CMU Grand Chanllenge Dataset. You can reproduce the same result using the function demoKit.m in the code.
Download the [Video 3MB].
Publications
-
[1]Generalized Time Warping for Multi-modal Alignment of Human MotionIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012
F. Zhou and F. De la Torre -
[2]Canonical Time Warping for Alignment of Human BehaviorAdvances in Neural Information Processing Systems (NIPS), 2009
F. Zhou and F. De la Torre[Paper 5MB] -
[3]Generalized Canonical Time WarpingIEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 38(2):279-294, 2016
F. Zhou and F. De la Torre[Paper 18MB]
References
-
[4]Fundamentals of Speech RecognitionPrentice Hall, 1993
L. Rabiner and B. Juang