Patient workflow in radiotherapy is one of the most complex workflows. There are many steps involved: choice of the radiotherapy treatment scheme; image acquisition of the patient in treatment position; segmentation of the target volumes and organs-at-risk (OAR) using multimodal imaging; treatment planning; delivery of treatment including monitoring of patient positioning, movements, and delivered dose; and finally, post-treatment follow-up. To facilitate and improve the efficiency of this workflow, artificial intelligence (AI) systems have been proposed for automatic organ segmentation, error prevention, or treatment planning [1,2]. However, these systems are still seldom used in clinical routines. For instance, manual delineation of target volumes and OAR is still the standard routine for most clinical centers, even though it is time consuming and prone to intra- and inter-observer variations [3]. One issue is the limited performance of current commercial software. In radiotherapy, toxic and fatal doses are sometimes delivered at 1 or 2 mm from risk organs; therefore, it is vital that segmentation is extremely accurate (see Fig. 1). However, current automatic segmentation software cannot achieve the necessary level of accuracy. Consequently, radiation oncologists may lose more time correcting automatically segmented structures than by manually segmenting the structures themselves. Deep learning (DL) is a branch of AI and machine learning, which has enjoyed considerable success in recent years in diverse fields including science, business, and government. DL has dramatically supplanted other machine learning methods for applications such as recognition and image processing in computer vision, by achieving human-equivalent performance on some tasks [4–۶]. DL techniques also open promising perspectives in AI applied to radiotherapy and may significantly improve the radiotherapy patient workflow in the coming years [7,8]. To illustrate the rapidly evolving interest aroused by these new techniques in radiotherapy, Fig. 2 shows the number of DL papers published in this field since 2012.