Evaluation Protocol
STS 2026 is organized into three tasks: metal artifact CBCT teeth segmentation, CBCT-IOS registration and MMDental multimodal analysis. Together, they cover robust CBCT tooth analysis, cross-modal dental geometry fusion and multimodal clinical reasoning.
Evaluation Tasks
| Task | Submission | Assessment |
|---|---|---|
| Task 1: Metal Artifact CBCT Teeth Segmentation | Tooth and root-canal segmentation masks for CBCT scans affected by metal artifacts. | Image- and instance-level Dice, mIoU and NSD plus Identification Accuracy (IA); the final Challenge_Score combines them. |
| Task 2: CBCT-IOS Registration | Rigid transformation matrices (upper and lower jaw) aligning the IOS crown surface to the CBCT volume. | Mean Translation Error (MTE, mm) and Mean Rotation Error (MRE, degree); lower is better. |
| Task 3: MMDental Multimodal Analysis | Structured seven-field dental reports generated from CBCT volumes and expert records. | Weighted composite of text-overlap metrics, FDI tooth-notation, diagnosis-code and treatment-plan agreement, and field completion. |
Metrics
- Task 1 — Dice, mIoU and NSD are computed at image level (whole dentition foreground) and instance level (per tooth / root-canal label); Identification Accuracy (IA) counts labels with IoU ≥ 0.5.
- Task 1 — Challenge_Score = 0.15 × image_score + 0.75 × instance_score + 0.10 × IA, where image_score is the mean of image-level Dice, mIoU and NSD and instance_score = 0.35 × Dice + 0.25 × mIoU + 0.40 × NSD at instance level. Cases containing metal artifacts are weighted × 2.
- Task 2 — Mean Translation Error (MTE, mm) and Mean Rotation Error (MRE, degree) between the predicted and reference rigid transformations, averaged over the upper and lower jaws; lower is better.
- Task 3 — Weighted_Score is a weighted combination of BLEU-1/2/3/4, ROUGE-L, METEOR, FDI tooth-notation precision/recall/F1, diagnosis-code recall, treatment-plan recall and report field-completion rate.
- Segmentation efficiency (running time, GPU memory) is not part of the 2026 ranking.
Task 1 keeps image-level and instance-level metrics separate because metal-artifact blooming mainly corrupts anatomical boundaries and small structures such as root canals, while overall image overlap may remain high. Instance-level metrics and IA therefore carry most of the weight in the final score.
Ranking Rules
- Task 1 ranks teams by Challenge_Score (higher is better).
- Task 2 ranks teams by registration accuracy, using MTE as the primary metric and MRE as the secondary metric (both lower is better).
- Task 3 ranks teams by Weighted_Score (higher is better).
- The official final ranking of all three tasks is determined by the Docker test evaluation on the hidden test set; the Codabench validation leaderboard is not used for the final ranking.
- Missing or failed predictions receive the worst possible score for the affected cases (for example Dice = 0 for Task 1, and fixed penalties of 1,000,000 mm / 180° for failed Task 2 pairs).
Dataset Split
| Split | Cases | Provided Data |
|---|---|---|
| Training (Labeled) | 40 | CBCT, IOS, segmentation masks and registration matrices. |
| Training (Unlabeled) | 219 | Raw CBCT and IOS data for semi-supervised learning. |
| Validation | 20 | Raw CBCT and IOS data; ground truth withheld for server evaluation. |
| Test | 100 | Raw CBCT and IOS data; hidden ground truth for final ranking. |
| Total | 379 | All cases include metallic restorations and metal artifacts. |
The labeled training set and test set share the same metal artifact severity stratification: 30% mild, 40% moderate and 30% severe. This mirrored distribution keeps the task definition stable and evaluates robustness to metal artifacts rather than unexpected domain shift.
Statistical Analysis
The organizers will estimate 95% confidence intervals using bootstrap analysis, compare top teams with paired Wilcoxon signed-rank tests, and report variability with standard deviation, interquartile range and box-and-whisker plots. Additional analyses will include artifact severity stratification, semi-supervised learning efficacy and inter-rater reliability by artifact subgroup.
The latest public materials for STS 2026 will also be linked from the official GitHub repository.