Recently, deep learning has brought revolutions to many mobile and embedded systems that interact with the physical world using continuous video streams. Although there have been significant efforts to reduce the computational overheads of deep learning inference in such systems, previous approaches have focused on delivering ‘best-effort’ performance, resulting in unpredictable performance under variable environments. In this paper, we propose a runtime control method, called DMS (Dynamic Model Scaling), that enables dynamic resource-accuracy trade-offs to support various QoS requirements of deep learning applications. In DMS, the resource demands of deep learning inference can be controlled by adaptive pruning of computation-intensive convolution filters. DMS avoids irregularity of pruned models by reorganizing filters according to their importance so that varying number of filters can be applied efficiently. Since DMS’s pruning method incurs no runtime overhead and preserves the full capacity of original deep learning models, DMS can tailor the models at runtime for concurrent deep learning applications with their respective resource-accuracy trade-offs. We demonstrate the viability of DMS by implementing a prototype. The evaluation results demonstrate that, if properly coordinated with system level resource managers, DMS can support highly robust and efficient inference performance against unpredictable workloads.

Introduction

In the past few years, deep learning has emerged as a stateof-the-art approach that provides highly robust and accurate inference capability for many intelligent systems and services [1]. In particular, convolutional neural networks (CNNs or ConvNets) [2]–[۴] have brought revolutions to computer vision applications [5]. Deep CNNs play as generic feature extractors for various visual recognition tasks such as image classification [3], object detection [6], semantic segmentation [7], and image retrieval [3]. Such visual recognition tasks are essential for many intelligent systems interacting with the physical world using continuous streaming of video inputs. Some examples are augmented reality wearables [8], camerabased surveillance, drones, autonomous vehicles [9], and live video analytics [5], to name a few. However, the size and complexity of deep learning models has been a major challenge for resource-constrained mobile and embedded devices, and there have been significant efforts to reduce the amount of computation of deep learning models either by compressing deep learning models at a modest loss of inference accuracy [10]–[۱۳] or by offloading inference workloads to custom accelerators [14]–[۱۶]. Although these approaches have demonstrated significant gains in performance and efficiency, their resource demands are predetermined at development stages, incurring unpredictable ‘best-effort’ performance in highly dynamic environments. For instance, when a person with a wearable cognitive-assistance device walks to a more crowded area, more objects needs to be classified, resulting in sudden increases of overall inference latency and energy consumption [8].