Deep Reinforcement Learning and Deep Neuroevolution to perform localization, segmentation, classification, and change on radiological images


Deep Reinforcement Learning and Deep Neuroevolution to perform localization, segmentation, classification, and change on radiological images



Clinical radiologists spend much of their work time reviewing and interpreting radiological images to inform clinical diagnoses. However, problems with overload, backlog, and burnout are not uncommon. AI technology can automate time-consuming and tedious tasks. But current solutions have shortcomings. For example, the types of clinical conditions addressed by AI algorithms from any single vendor tend to be limited. Also, the big data requirements for developing such AI algorithms often detract from their versatility, accuracy, and portability across clinical contexts – after all, when large datasets are not available, the resulting algorithms may lack accuracy or the ability to be broadly deployed. Such algorithms may also lack the ability to undergo continuous learning and meta-learning to easily adapt to new institutions, populations, or even different scanner settings.

MSK’s novel invention overcomes current limitations by utilizing deep reinforcement learning and deep neuroevolution to enable a “small data” and noise-resistant approach in AI algorithm development.  MSK findings show that these algorithms can learn effectively from very small data sets—only a few dozen images. Further, our results suggest robust generalization to new institutions’ data in clinical deployment scenarios.


  • Increased flexibility – The ability to use smaller datasets unlocks faster development of robust predictive algorithms, requires far fewer annotated images, and is applicable in a wide variety of clinical conditions. The MSK team has validated this in instances including rare diseases for which large datasets simply do not exist.
  • Improved portability – Models developed in one clinical context may be easily ported and adapted to new clinical contexts. For example, an algorithm developed by one medical center may be re-deployed for clinical use elsewhere with minimal additional algorithm training. Additionally, any required transfer learning is made simpler and more seamless by this MSK method.
  • Greater agility – Reduced data preparation for small datasets will enable radiologists themselves to design, build, train, and deploy AI algorithms, all in a code-free and non-clinically obtrusive manner, facilitating expanded use in patient care and research settings.


CB Insights estimates that the AI diagnostic imaging solutions market will reach nearly $265B by 2026. The U.S. market consists primarily of academic-research institutions (20%) and private radiology practices (80%).


  • Stember JN, Shalu H. Deep Neuroevolution Squeezes More out of Small Neural Networks and Small Training Sets. arXiv preprint arXiv:2112.12990. 2021 Dec 24  Link
  • Stember J, Young R, Shalu H. Direct evaluation of progression or regression of disease burden in brain metastatic disease with Deep Neuroevolution. arXiv preprint arXiv:2203.12853. 2022 Mar 24 Link.
  • Stember JN, Shalu H. Deep reinforcement learning with automated label extraction from clinical reports accurately classifies 3D MRI brain volumes. Journal of Digital Imaging. 2022 May 13:1-0. Link
  • Stember J, Shalu H. Deep Reinforcement Learning Classification of Brain Tumors on MRI. In Innovation in Medicine and Healthcare 2022 (pp. 119-128). Springer, Singapore. Link




PCT/US2021/053508 filed October 5, 2021;  Provisional Nos. 63/087,475, 63/129,999, 63/145,678, and 63/237,804 pending


Joseph Stember, PhD, Assistant Attending, Department of Radiology, MSK


Kiana Williams, Digital Ventures Lead, Technology Transfer

Email: [email protected]

Stage of Development