:::info
This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Zhihang Ren, University of California, Berkeley and these authors contributed equally to this work (Email: peter.zhren@berkeley.edu);
(2) Jefferson Ortega, University of California, Berkeley and these authors contributed equally to this work (Email: jefferson_ortega@berkeley.edu);
(3) Yifan Wang, University of California, Berkeley and these authors contributed equally to this work (Email: wyf020803@berkeley.edu);
(4) Zhimin Chen, University of California, Berkeley (Email: zhimin@berkeley.edu);
(5) Yunhui Guo, University of Texas at Dallas (Email: yunhui.guo@utdallas.edu);
(6) Stella X. Yu, University of California, Berkeley and University of Michigan, Ann Arbor (Email: stellayu@umich.edu);
(7) David Whitney, University of California, Berkeley (Email: dwhitney@berkeley.edu).
:::
Table of Links
Abstract and Intro
Related Wok
VEATIC Dataset
Experiments
Discussion
Conclusion
More About Stimuli
Annotation Details
Outlier Processing
Subject Agreement Across Videos
Familiarity and Enjoyment Ratings and References
9. Outlier Processing
We assessed whether there were any noisy annotators in our dataset by computing each individual annotator’s agreement with the consensus. This was done by calculating the Pearson correlation between each annotator and the leaveone-out consensus (aggregate of responses except for the current annotator) for each video. Only one observer in our dataset had a correlation smaller than .2 with the leave-oneout consensus rating across videos. We chose .2 as a threshold because it is often used as an indicator of a weak correlation in psychological research. Importantly, if we compare the correlations between the consensus of each video and a consensus that removes the one annotator who shows weak agreement, we get a very high correlation (r = 0.999) indicating that leaving out that subject does not significantly influence the consensus response in our dataset. Thus, we decided to keep the annotator with weak agreement in the dataset in order to avoid removing any important alternative annotations to the videos.
:::info
This paper is available on arxiv under CC 4.0 license.
:::