​Gemma ROIG (University of Frankfurt)  

Short abstract:

Deep Neural Networks (DNNs) are state-of-the-art models for many vision tasks. We propose an approach to assess the relationship between visual tasks and their task-specific models. Our method uses Representation Similarity Analysis (RSA), which is commonly used to find a correlation between neuronal responses from brain data and models. With RSA we obtain a similarity score among tasks by computing correlations between models trained on different tasks. We demonstrate the effectiveness and efficiency of our method to generating task taxonomy on the Taskonomy dataset. Also, results on PASCAL VOC suggest that initializing the models trained on tasks with higher similarity score show higher transfer learning performance. Finally, we explore the power of DNNs trained on 2D, 3D, and semantic visual tasks as a tool to gain insights into functions of visual brain areas (early visual cortex (EVC), OPA and PPA). We find that EVC representation is more similar to early layers of all DNNs and deeper layers of 2D-task DNNs. OPA representation is more similar to deeper layers of 3D DNNs, whereas PPA representation to deeper layers of semantic DNNs. We extend our study to performing searchlight analysis using such task specific DNN representations to generate task-specificity maps of the visual cortex. Our findings suggest that DNNs trained on a diverse set of visual tasks can be used to gain insights into functions of the visual cortex. This method has the potential to be applied beyond visual brain areas.