What is EgoTaskQA?

Understanding human tasks through video observations is an essential capability of intelligent agents. The challenges of such capability lie in the difficulty of generating a detailed understanding of situated actions, their effects on object states (i.e., state changes), and their causal dependencies. These challenges are further aggravated by the natural parallelism from multi-tasking and partial observations in multi-agent collaboration. We introduce the EgoTaskQA benchmark for evaluating models' capabilities on these critical aspects in egocentric videos.

We design questions that target the understanding of (1) action dependencies and effects, (2) intents and goals, and (3) agents' beliefs about others. We consider reasoning problems including descriptive (what status?), predictive (what will?), explanatory (what caused?), and counterfactual (what if?) queries to provide diagnostic analyses on spatial, temporal, and causal understandings of goal-oriented tasks.

Code

View codes and instructions for reproducing the experiments on Github.

Bibtex

@inproceedings{jia2022egotaskqa,

title = {EgoTaskQA: Understanding Human Tasks in Egocentric Videos},

author = {Jia, Baoxiong and Lei, Ting and Zhu, Song-Chun and Huang, Siyuan},

booktitle = {The 36th Conference on Neural Information Processing Systems (NeurIPS 2022) Track on Datasets and Benchmarks},

year = {2022}

}

Team

¹ UCLA Center for Vision, Cognition, Learning, and Autonomy (VCLA)

² Beijing Institute for General Artificial Intelligence (BIGAI)

³ Institute for Artificial Intelligence Peking University

⁴Department of Automation, Tsinghua University