Nanoparticles of Mn0.03-xCoxZn0.97O (x = 0.00, 0.01, 0.02, 0.03) have been synthesized using sol-gel technique. The wurtzite structure of ZnO has been confirmed and no impurity phase has been detected in any of the nanoparticle samples. The compositional study exhibits the presence of transition metal (TM) and further X-ray absorption near edge spectroscopy indicates quite low presence of desired Mn2+ ions whereas the presence of desired Co2+ ions is substantial. Morphological patterns have confirmed the formation of nanoparticles and supported the systematic variation of crystallite size as observed from structural study. The estimated band gap exhibits a systematic trend of red shift associated with weak quantum confinement effect and it is commensurate with the variation of particle size. Point defects particularly zinc interstitial (I-Zn) has been detected from photoluminescence (PL) measurement. Field dependent magnetization (M-H) measurement exhibits room temperature (RT) ferromagnetic (FM) ordering. The temperature dependent magnetization (M-T) measurements demonstrate the presence of simultaneous antiferromagnetic coupling which is responsible for weakening of FM ordering. Moderate temperature (350 degrees C) annealing has been found to be very effective for achieving RT FM ordering in TM doped ZnO nanoparticles and Co seems to be quite efficient than Mn so far as substitutional replacement of Zn by TM ion is concerned.