مقاله انگلیسی رایگان در مورد معیارهای ارزیابی برای مولدهای تصویر کهکشانی – الزویر 2023

Abstract

     A major problem with deep generative models is verifying that the generated distribution resembles the target distribution while the individual generated sample is indistinguishable from the original data. In particular, for application in astrophysics we need to be sure that the generated data matches our prior knowledge and that the generated samples entail all object types with the correct frequency and diversity. We currently lack objective ways to systematically assess these quality aspects, where human inspection reaches its limits, as this requires detailed analysis of a large data volume. In this work, we identify reasonable metrics for the quality of galaxy image generators. To this end, we compare a small set of conditional image generators, trained on galaxy images with classification labels for visual morphology features. Our main contribution is a new set of cluster-based metrics for matching the generated distribution to the target distribution. Furthermore, we use the Wasserstein distance on proxies for galaxy morphology as well as a number of other metrics commonly used for image generators. The newly introduced cluster-based metrics are good proxies for the quality of the generated distribution and are suited for automatized identification of mode collapse. Furthermore, the cluster metrics allow for a qualitative interpretation of the generated distribution. The metrics based on morphological statistics provide a useful tool to probe the physical soundness of generated samples. Finally, we find that kernel inception distance used with an InceptionV3 model pre-trained on ImageNet is a good proxy for the overall quality of galaxy image generators, although it cannot be interpreted that easily.

Introduction

     Upcoming astronomical surveys with telescopes, such as Euclid (Laureijs et al., 2012) and LSST (Abell et al., 2009), will provide a wealth of data with billions of galaxy images. These hold unprecedented insight in highly researched astrophysical and cosmological questions, such as formation and evolution of galaxies, the cosmic distribution of dark matter, as well as the expansion history of the Universe (Laureijs et al., 2011). However, the huge amount of images is far too big to be investigated by astrophysicists on an individual basis. Instead, fast and systematic extraction of galaxy properties from their images is required in order to inform and constrain physical models. There are a number of computational tools that have been developed and are already in use by the astrophysical community (Rodriguez-Gomez et al., 2019, Boquien et al., 2019, Shamir, 2011). Though these allow for a systematic extraction of properties, they require too much computational resources to provide big collections of high quality mock images in a reasonable amount of time. Hence, it is necessary to replace these tools by faster methods, e. g. by using machine learning techniques, such as deep neural networks (Lovell et al., 2019, Ferreira et al., 2020, Walmsley et al., 2022 e. g.).

Conclusions

     In order to find suitable evaluation metrics to assess the quality of galaxy image generators, we investigate a number of evaluation metrics with the potential to measure different quality aspects. These metrics are probed with a small number of conditional generative models, some of which purposely are of worse quality than the others. We identified evaluation metrics that are good proxies for the quality of individual images and the resemblance of the target distribution. Our main results are:

• The newly introduced cluster-based metrics (Section 3.1) are a formidable new tool to assess the distribution of generated data.

• The KID metric on features of the pre-trained InceptionV3 model (Szegedy et al., 2015) provides a useful proxy on the overall quality of galaxy image generators.

• Using an ALReLU activation (Mastromichalakis, 2020) significantly enhances classification accuracy for rare object types. This is required to train conditional generators on the highly imbalanced dataset of galaxy images.

     We introduce a new set of cluster-based metrics (Section 3.1), which are good proxies for the distribution quality. In particular, they allow for a qualitative interpretation regarding the sample diversity and the amount of samples generated for different types. Thus, they provide an intuitive new tool to assess the resemblance of the target distribution in a qualitative way, which so far has been a rather illusive task. In addition, one of the cluster metrics, the cluster error provides a formidable tool to identify mode collapse. Moreover, the cluster metrics have the potential to be transformed into a loss function to train generative models to reproduce the distribution of training data exactly, or any other target distribution, e. g. balanced datasets. Together, the aforementioned metrics provide a good basis to assess the quality of trained image generators, which is not necessarily limited to the context of galaxy images. We are currently working on providing losses based on these cluster metrics, which have the potential to drastically improve training of generative models regarding distribution quality, which is especially valuable for GANs.