The court said the massacre of Bosnian Muslim men in Srebrenica did constitute genocide, but could not be directly attributed to the Belgrade government.
It found, however, that Serbia had violated international law by failing to prevent the killings.
Bosnia had asked the court to rule on whether Serbia committed genocide during the 1992-95 war.
It was the first time a state had been tried for genocide, which was outlawed in a UN convention in 1948 after the Nazi Holocaust.
The International Criminal Tribunal for the former Yugoslavia had already ruled that the Srebrenica massacre was an act of genocide, and has convicted two former Bosnian Serb military officers for aiding and abetting the killings.
Reading from a lengthy judgment before pronouncing the final ruling, ICJ president Judge Rosalyn Higgins said the court concluded that the acts committed at Srebrenica fell within the relevant articles of the genocide convention.
She added that in other mass killings of Bosnian Muslims, the court was not convinced the perpetrators had the specific intent to commit genocide.
Earlier in the ruling she said the court found it established that Serbia was making considerable military and financial support available to the Bosnian Serbs.
