As an increasing number of intelligent devices access networks wirelessly, device-free Wi-Fi-based localization has garnered increasing attention due to its privacy preservation and low cost. By collecting the Channel State Information (CSI) from Wi-Fi devices and representing it as images, deep neural networks can be trained to extract deep spatial features related to human positions from the constructed CSI images. This enables the learning of complex mappings from wireless signal perturbations to specific physical locations or motion patterns, thereby achieving high-accuracy localization without requiring any carried device. However, as environments change over time, the collected CSI image fingerprints may shift, leading to degraded localization performance. Although recollecting labeled data in the altered environment can address this issue, it is undoubtedly time-consuming and labor-intensive. In this paper, based on the Vision Transformer (ViT) network, we employ a deep subdomain adaptation network to align fine-grained information of CSI images within each location category before and after environmental changes. This method effectively bridges the gap between unlabeled CSI images from the changed environment and the labeled fingerprint database, thereby enhancing the robustness of device-free Wi-Fi localization.