The current infrared image pedestrian detectors have problems with high rates of false positives and false negatives. To solve these problems, we proposed an improved anchor-free fully convolutional one-stage object detection (FCOS) algorithm. Firstly, we introduced the channel attention module squeeze excitation (SE)-Block in the FCOS backbone network, which was used to learn how to model the relative importance between different feature channels, and to achieve the weight recalibration of the features extracted from the convolution neural network, and improve the weight values that are more important for pedestrian target detection. Secondly, soft non-maximum suppression (Soft-NMS) replaced the conventional NMS within the algorithm’s post-processing phase, which was used to reduce the probability of missed detection for occluded pedestrians. The experimental results show that our improved FCOS algorithm improves the average precision (AP) by 6.71% on the original dataset and 7.97% on the augmented KAIST pedestrian dataset compared with the original FCOS algorithm. Our improvements effectively meet the real-time requirements and there is no significant decrease in speed compared with the original FCOS algorithm, and decreased the false positives and false negatives for infrared image pedestrian detection.