Improved ground truth annotation by multimodal image registration from 3D ultrasound to histopathology for resected tongue carcinoma

Objectives: This study’s objectives are (1) to investigate the registration accuracy from intraoperative ultrasound (US) to histopathological images, (2) to assess the agreement and correlation between measurements in registered 3D US and histopathology, and (3) to train a nnUNet model for automatic segmentation of 3D US volumes of resected tongue specimens. Methods: Ten 3D US volumes were acquired, including the corresponding digitalized histopathological images (n = 29). Based on corresponding landmarks, the registrations between 3D US and histopathology images were calculated and evaluated using the target registration error (TRE). Tumor thickness and resection margins were measured based on three annotations: (1) manual histopathological tumor annotation (HTA), manual 3D US tumor annotation, and (2) the HTA registered in the 3D US. The agreement and correlation were computed between the measurements based on the HTA and those based on the manual US and registered HTA in US. A deep-learning model with nnUNet was trained on 151 3D US volumes. Segmentation metrics quantified the model’s performance. Results: The median TRE was 0.42 mm. The smallest mean difference was between registered HTA in US and histopathology with 2.16 mm (95% CI − 1.31; 5.63) and a correlation of 0.924 (p < 0.001). The nnUNet predicted the tumor with a Dice similarity coefficient of 0.621, an average surface distance of 1.15 mm, and a Hausdorff distance of 3.70 mm. Conclusion: Multimodal image registration enabled the HTA’s registration in the US images and improved the agreement and correlation between the modalities. In the future, this could be used to annotate ground truth labels accurately.