An improved algorithm for traffic sign detection based on YOLOv8 is proposed. Firstly, YOLOv8n is used as the base model of the network, the inverted residual mobile block and exponential moving average (iRMB_EMA) attention mechanism is used to improve the model’s ability to perceive small targets, which reduces the leakage detection phenomenon of the model, convolution (Conv) is upgraded to receptive-field attention convolution (RFAConv), which improves the model’s ability to deal with details and complexity in the image, the idea of adaptive spatial feature fusion (ASFF) is introduced in the detection head, and the small target detection layer, a four-head detection head is designed to improve the model’s ability to detect small targets, solves the case of feature loss due to cross-scale fusion, and use of the Inner-minimum points distance intersection over union (MPDIoU) loss function provides a more accurate loss metric by calculating the distance of key points between the predicted and true frames. The experimental results of this algorithm on the public dataset CCTSDB show that the improved model mean average precision (mAP) reaches 82.6%, which is 4% higher than the YOLOv8n. The experimental results of dataset TT100k show that the mAP reaches 84.5%, which is 7.1% higher than the YOLOv8n. This algorithm effectively improves the problem of detail perception and leakage of the model in the detection of small targets, and has a significant detection effect compared to other algorithms.