Formation of Ge nanocrystals in SiNx matrices has been studied using plasma enhanced chemical vapor deposition in both as deposited samples as well as in post-vacuum annealed samples. Low temperature and short duration anneals in vacuum resulted in Ge nanocrystals whereas prolonged anneals at higher temperatures resulted in Ge nanocrystals accompanied with SiGe formation at the SiNx/Si interface. Raman Scattering Spectroscopy was extensively used to track the formation of various phonon modes during the diffusion of Ge through SiNx and into the Si substrate.