More accurate segmentation of skin cancers in dermoscopy images is crucial for clinical treatment. However, the prevalence of interfering noise in dermoscopy images poses a challenge to its accurate segmentation. For this reason, this paper proposes an improved GLF-Segformer to improve segmentation. The model adds polarized self-attention (PSA) module and R-convolution and attention fusion module (R-CAFM) to the Segformer’s encoder to enhance the ability to capture local information and facilitate the effective fusion of local and global information. The decoder employs an innovative two-stage hybrid up-sampling to effectively reduce information loss. In addition, a new hybrid loss function is designed to further improve the segmentation accuracy of the model at complex boundaries. The experimental results show that GLF-Segformer achieves 90.73% and 89.85% mean intersection over union (mIoU) on two standard datasets, ISIC2017 and ISIC2018, respectively, and exhibits better segmentation performance compared to other comparison algorithms.