By doping with Mg atoms, the bandgap of Mg-doped NiO thin films can be adjusted larger. By using NiO and MgO as sputtering targets, Mg-doped NiO thin films were deposited using radio-frequency magnetron co-sputtering method in pure argon and pure oxygen gas, respectively. The crystal structure, morphological characteristics, composition and optical properties of the obtained films were compared by X-ray diffraction (XRD), scanning electron microscope (SEM), energy dispersive spectrometer (EDS) and ultraviolet (UV)-visible spectrophotometer. The properties of the thin films deposited in different sputtering gases are quite different. For the films deposited in pure argon gas, it is a polycrystalline thin film with (200) preferred orientation, while the film deposited in pure oxygen has no preferred orientation. The grain size, molar ratio of Mg to Ni atoms and optical bandgap are larger for the films deposited in pure argon gas than those deposited in oxygen gas.