Article intro - Surgical Data Science Metrics Reloaded

There is a great initiative ongoing to define/redefine meaningful and accurate metrics for image-based procedures, supported by automated software methods. The core document is a massive white paper, allowing to fully submerge into the domain, from Lena Maier-Hein et al. 

Abstract

The field of automatic biomedical image analysis crucially depends on robust and meaningful performance metrics for algorithm validation. Current metric usage, however, is often ill-informed and does not reflect the underlying domain interest. Here, we present a comprehensive framework that guides researchers towards choosing performance metrics in a problem-aware manner. Specifically, we focus on biomedical image analysis problems that can be interpreted as a classification task at image, object or pixel level. The framework first compiles domain interest-, target structure-, data set- and algorithm output-related properties of a given problem into a problem fingerprint, while also mapping it to the appropriate problem category, namely image-level classification, semantic segmentation, instance segmentation, or object detection. It then guides users through the process of selecting and applying a set of appropriate validation metrics while making them aware of potential pitfalls related to individual choices. In this paper, we describe the current status of the Metrics Reloaded recommendation framework, with the goal of obtaining constructive feedback from the image analysis community. The current version has been developed within an international consortium of more than 60 image analysis experts and will be made openly available as a user-friendly toolkit after community-driven optimization.

Source: Maier-Hein, Lena; et al; Menze, Bjoern (2022). Metrics reloaded: Pitfalls and recommendations for image analysis validation. arXiv.org 2206.01653, University of Zurich