Metrics

This page provides comprehensive definitions of segmentation metrics used in RankSEG, including Dice, IoU, and Accuracy. Understanding these metrics is essential for choosing the right optimization strategy for your segmentation task.

Definitions

In brief, the definitions of Dice and IoU optimized in rankseg are consistent with per-image level Dice and IoU in [2] or aggregation_level='samplewise' in TorchMetrics [1].

Dice Coefficient

The Dice coefficient, also known as the F1 score or Sørensen-Dice coefficient, measures the overlap between predicted and ground-truth segmentations. It is the harmonic mean of precision and recall.

The per-image level averaged Dice score for class \(c\) across all samples \(i\) is computed as:

\[\boxed{\text{Dice}_c = \frac{1}{n} \sum_{i=1}^{n} \frac{2 \cdot \text{TP}_{ic} + \gamma}{2\text{TP}_{ic} + \text{FP}_{ic} + \text{FN}_{ic} + \gamma} = \frac{1}{n} \sum_{i=1}^{n} \frac{2 \cdot |\hat{\mathbf{y}}_{ic} \cap \mathbf{y}_{ic}| + \gamma}{|\hat{\mathbf{y}}_{ic}| + |\mathbf{y}_{ic}| + \gamma}}\]

Where \(\text{TP}_{ic}\), \(\text{FP}_{ic}\) and \(\text{FN}_{ic}\) denote the number of true positive, false positive and false negative pixels for class \(c\) in sample \(i\), respectively.

IoU

The IoU (Intersection over Union), also known as the Jaccard Index, measures the ratio of intersection to union between predicted and ground-truth segmentations.

The per-image level averaged IoU score for class \(c\) across all samples \(i\) is computed as:

\[\boxed{\text{IoU}_c = \frac{1}{n} \sum_{i=1}^{n} \frac{ \text{TP}_{ic} + \gamma}{\text{TP}_{ic} + \text{FP}_{ic} + \text{FN}_{ic} + \gamma} = \frac{1}{n} \sum_{i=1}^{n} \frac{|\hat{\mathbf{y}}_{ic} \cap \mathbf{y}_{ic}| + \gamma}{|\hat{\mathbf{y}}_{ic} \cup \mathbf{y}_{ic}| + \gamma}}\]

Where \(\text{TP}_{ic}\) and \(\text{FP}_{ic}\) denote the number of true positive and false positive pixels for class \(c\) in sample \(i\), respectively.

Properties

• Range: Both Dice and IoU have range \([0, 1]\), where 1 indicates perfect overlap and 0 indicates no overlap.

• Smoothing: \(\gamma\) (controlled via smooth parameter) prevents division by zero. A small value of smooth is added to the numerator and denominator of both metrics. This helps to avoid division by zero and mitigates the effect of classes with very few positive pixels.

Note

Since the PASCAL VOC 2008 Challenge [3], Dice and IoU metrics have typically been computed at per-dataset level. However, this approach aggregates the confusion matrix across all images, causing large objects (with more pixels) to dominate the results over smaller objects [2]. Therefore, we believe using per-image level metrics is more appropriate for evaluating segmentation.

Dice and IoU metrics optimized in rankseg follows a samplewise aggregation strategy, which matches the aggregation level used in TorchMetrics [1] (aggregation_level='samplewise'). The per-image level Dice and IoU are denoted as \(\text{Dice}_c^C\) and \(\text{IoU}_c^C\) in [2].

Moreover, for aggregation over multiple classes, rankseg supports optimization of both \((\text{mDice}^I, \text{mIoU}^I)\) and \((\text{mDice}^C, \text{mIoU}^C)\) in [2], see definitions (4)-(8) in [2].

References

[1] (1,2)

torchmetrics.segmentation.DiceScore

[2] (1,2,3,4,5)

Wang, Zifu, etal. “Revisiting evaluation metrics for semantic segmentation: Optimization and evaluation of fine-grained intersection over union.” Advances in Neural Information Processing Systems 36 (2023): 60144-60225. paper link

[3]

Everingham, Mark, et al. “The pascal visual object classes challenge: A retrospective.” International Journal of Computer Vision 111.1 (2015): 98-136.