Advanced Multi-Scale Vision Transformer with Neural Architecture Search for Thyroid Cancer Diagnosis Using Ultrasound Images

Thyroid cancer diagnosis from ultrasound images requires accurate classification into four risk categories: Not Suspicious, Mildly Suspicious, Moderately Suspicious, and Highly Suspicious. The task is challenging due to subtle visual differences between classes and severe data imbalance. This work proposes two novel deep learning architectures for automated thyroid cancer risk stratification. DBHNet is a dual‑branch hybrid network that combines a ResNet50 branch for local texture extraction and a DeiT‑Small transformer branch for global contextual reasoning, fused through learnable weighted concatenation. PyCAFNet is a pyramid cross‑scale attention network built on an EfficientNet‑B4 backbone, incorporating sequential cross‑scale attention modules and a NAS‑inspired weighted multi‑scale fusion head. Both models are benchmarked against MSViTAFF, an existing multi‑scale Vision Transformer baseline. Class imbalance is addressed using an augmentation pipeline that includes CLAHE, elastic transforms, and geometric augmentations. Explainability is embedded through Grad‑CAM for PyCAFNet and RISE combined with attention rollout for DBHNet, providing clinically interpretable heatmap visualizations. DBHNet is deployed in a Flask‑based web application with relevance gating to reject non‑thyroid inputs, enabling rapid risk prediction.

Keywords: Thyroid cancer diagnosis · Ultrasound image classification · Dual-branch hybrid network · Pyramid cross-scale attention · Neural architecture search · Explainable AI (XAI) · RISE · Attention Rollout · Grad-CAM · Flask deployment