In the present study, the structural, optical, and magnetic properties of (ZnSn)_1-xCo_xO nanocomposites (NCs) were reported. The transmission electron microscope (TEM) images of ZnSnO NCs showed mixed nanorods and nanosheets morphologies that completely transformed to mostly nanocubes with Co-ions incorporation. The optical band gap (E_g) of nanostructured ZnO, SnO, and ZnSnO NCs was 3.2, 4.5, and 3.9 eV, respectively. The estimated E_g value of all (ZnSn)_1-xCo_xO NCs was lower than the undoped ZnSnO NCs. The nanostructured ZnO revealed poor room-temperature ferromagnetic (RTFM) behavior whereas nanostructured SnO exhibited paramagnetic behavior at low magnetic field strength followed by diamagnetic behavior at high magnetic field strength. The (ZnSn)_1-xCo_xO NCs exhibited strong improvement in the RTFM, where the hybrid (ZnSn)₆₀Co₄₀O NCs exhibited the highest saturation magnetization (M_s) of 2 emu·g⁻¹. This value was 5-times higher than the undoped ZnSnO NCs and 2-orders of magnitude higher than the pure ZnO phase. However, with increasing Co-content up to 50 %, the M_s was reduced to 1.16 emu·g⁻¹ demonstrating the corruption of RTFM. The present study indicated the importance of composition utilization for tuning physical properties of (ZnSn)_1-xCo_xO NCs and improving the performance of these NCs in optoelectronic and spintronics applications.