We propose a novel deep multibranch and multitask neural network for artist, style, and genre painting categorization. The multibranch approach allows us to exploit at the same time the coarse layout of the painting and the fine-grained structures by using painting crops at different resolutions that are wisely extracted using a Spatial Transformer Network trained to identify the most discriminative subregions of paintings. The effectiveness of the proposed network is proved in experiments that are performed on a new dataset originally sourced from wikiart.org and hosted by Kaggle, and made suitable for artist, style and genre multitask learning. The dataset here proposed and made available for research is named MultitaskPainting100k, and is composed by 100K paintings, 1508 artists, 125 styles and 41 genres annotated by human experts. Among the different variants of the proposed network, the best method achieves accuracy levels of 56.5%, 57.2%, and 63.6% on the MultitaskPainting100k dataset for the tasks of artist, style and genre prediction respectively.

Introduction

Automatic categorization and retrieval of digital paintings is gaining increasing attention due to the large quantities of visual artistic data made available by art museums that have digitized or are digitizing their artworks (Carneiro et al., 2012; Mensink & Van Gemert, 2014; Khan et al., 2014; Mao et al., 2017). In this work, we deal with the problem of categorizing paintings by automatically predicting the artist who painted them (e.g. Monet, van Gogh, etc.), the pictorial styles (e.g. Impressionism, Baroque, etc.), and the genres (e.g. portrait, landscape, etc.) (Anwer et al., 2016). These three tasks are very challenging due to the large amount of both inter- and intra-class variations: in fact there are different personal styles in the same art movement, and the same artist may have drawn in one or more different pictorial styles and genres. To have an idea of the difficulty of these tasks some examples taken from the dataset used in this work (i.e. MultitaskPainting100k) are reported in Figure 1. Artist classification consists in automatically associating the painting to its painter. In this task factors such as stroke patterns, the color palette used, the scene composition, and the subject depicted must be taken into account (Fichner-Rathus, 2011). Style classification consists in automatically assigning a painting into the school or art movement it belongs to.