Denmark has rejected a government proposal to deepen the EU member's participation in the bloc's justice cooperation, Prime Minister Lars Lokke Rasmussen said.
"It is a clear no... I have full respect for the Danes' decision," he said at a press conference after the referendum.
Mr Rasmussen had campaigned for the Yes side which had advocated for international coordination in the fight against cross-border crime, including violent extremism.
The No camp was led by the anti-EU, anti-immigration Danish People's Party (DPP) which believes dropping Denmark's justice 'opt-out' would have given too much power to Brussels and risked leading to more immigration.
"It is my clear impression that it's not so much what we have voted about that the Danes have turned their backs on, but perhaps what we haven't voted on," Mr Rasmussen said.
The result reflected "insecurity and uncertainty" over the consequences of a Yes vote and "maybe also general EU scepticism,” he said.
Denmark does not fully participate in the EU's justice and home affairs policies after Danish voters - wary of the "ever closer union" - rejected the Maastricht Treaty in 1992.
Copenhagen was then granted opt-outs on several EU policy areas, including justice, and Danes then said Yes to Maastricht in 1993.
With all the votes counted, the No camp won 53.1% against 46.9% to the Yes camp with a turnout of 72%, which was higher than expected.
"The Danes know that when things are left to Brussels, they're left a long way away in a non-transparent system where we lose a lot of our democracy," the Danish People's Party leader Kristian Dahl Thulesen said after most of the votes had been counted.
Such sentiment reflects a growing scepticism within the 28-member bloc as Brussels struggles to deal with problems ranging from Greece teetering on bankruptcy, a massive refugee crisis and the spread of attacks linked to Islamic militants.
The No victory will cheer Britain's UK Independence Party, which wants a total withdrawal, or a "Brexit", from the EU as well as other far-right factions such as the French National Front leader Marine Le Pen.
Danes were told by the government that certain EU laws were needed to keep the country within the cross-border police agency, Europol.
But instead of seeking approval for the 22 EU acts slated for adoption, Danes were instead asked to entrust to parliament the power to decide on such opt-ins.
That, analysts said, made it easy for the No camp to play on Danish distrust of politicians.
"It has been easy to create insecurity about what would happen with a Yes vote because what was on the menu was giving parliament a wider frame to involve Denmark in the EU," Aarhus University Professor Rune Stubager said.