About 90 women have reported being robbed, threatened or sexually molested at the New Year's celebrations outside Cologne's cathedral by young, mostly drunk, men, police said.
One woman said she was raped.
Police and witnesses have said about 1,000 men, split into groups, attacked women in the square.
Some witnesses said the men were of North African appearance, stirring strong emotions in Germany, where Chancellor Angela Merkel has welcomed people fleeing war zones in the Middle East and Africa.
While politicians urged people not to place refugees under blanket suspicion, the incident, described by Cologne police as a "new dimension of crime", fuelled calls from right-wing groups to halt admissions of migrants and refugees.
Germany took in just over a million last year, far more than any other European country.
Witnesses and women who reported being attacked described the men as looking as if they were from "the Arab or North African region", said police chief Wolfgang Albers, who added that they were mostly between 18 and 35 years old.
Cologne mayor Henriette Reker said there was no reason to believe that people involved in the attacks were refugees.
"We will not accept the disgusting attacks on women. All perpetrators must be held to account," said Justice Minister Heiko Maas, a Social Democrat, in a tweet.
The right-wing Alternative for Germany (AfD), which has gained in polls in part at Ms Merkel's expense thanks to a campaigna gainst refugees, said she should close the border.
"Mrs Merkel, is Germany 'colourful and cosmopolitan' enough or you after the wave of crimes and sexual attacks?" tweeted AfD chief Frauke Petry.
Ms Merkel has urged people to respect strangers.
"We .. respect everyone, even if we don't know them," she said. "That is the case not only for Germans, but for everyone."
There are almost daily attacks on refugee shelters.
"Events like that in Cologne foster xenophobia," Roland Schaefer, head of Germany's association of towns and localities, told reporters.
Integration commissioner Aydan Ozoguz warned against refugees and foreigners being put under "blanket suspicion".
After a crisis meeting, Ms Reker said the perpetrators had acted "absolutely unacceptably" and women must continue to join celebrations but with more protection.
Some politicians have suggested installing more security cameras and increasing police numbers at traditional carnival celebrations next month when the city closes down for five days of drunken street parades and parties.
Ms Reker was stabbed in the neck and seriously hurt in October, just a day before she was elected mayor. Police said that attack appeared to be motivated by her support for refugees.