Employing deep learning for pavement crack detection can significantly enhance accuracy, and federated learning can help to overcome the challenges of data silos and data security. This paper proposes a lightweight method to detect cracks. We use lightweight encoder modules to extract multi-scale features, and further feature fusion and modelling by selective fusion blocks and irregular-aware blocks. Moreover, this method combines federated learning with crack detection, effectively resolving the tension between data privacy and data sharing in decentralized devices. Experiments were conducted to compare our proposed method with other crack detection methods on two publicly available datasets, including the original method without federated learning and five state-of-the-art methods combined with the same federated learning framework. The experiment proves that our model obtains comparable F1 score with minimal parameters and computational effort. For example, compared to the original method, the number of parameters and computation are reduced by 67.1% and 73.6%, respectively.