Taoiseach Leo Varadkar has said it was unrealistic to expect the backstop to avoid a hard border to apply to the UK as a whole.
He also said the UK could not avail of the EU single market for goods and not services, saying this could ultimately "destroy" the European Union.
Speaking at the end of a two-day EU summit, Mr Varadkar said the EU was willing to make concessions on Northern Ireland remaining in the customs union and single market for goods as it was a "small place", but such a proposal would not be on offer for the UK as a whole.
His comments are the most forthright rejection of the UK's backstop offer of 7 June, which would foresee the UK as a whole remaining in the customs union and single market for a temporary period, in order to avoid a hard border on the island of Ireland, and to avoid any checks on the Irish Sea.
The Taoiseach also rejected any prospect of the UK remaining in the EU single market for goods, but diverging from the single market services, an idea which has recently gained currency in the UK.
"For an economy the size of the UK, that's not a realistic proposal," he said.
"Bear in mind what that would involve. It would involve the UK essentially having the benefit of one of the four freedoms, which is having access to the free market for goods, but not services and not capital ... and not free movement of labour.
"If that principle were to be conceded I'm sure there would be eurosceptic and rightwing and populist parties in every second country in Europe who would say, well, we want the same deal.
"That would be the beginning of the end of the single market, and while we really regret the UK has decided to leave the European Union, we're not going to let them destroy the European Union."
Mr Varadkar said the British Prime Minister Theresa May had briefed him on the broad outline of the forthcoming White Paper on Britain's future relationship with the EU, but he would not comment on its merits until he had seen the paper himself.
"The optimist in me would love to see the UK coming forward with a White Paper which allows the EU and UK to have a relationship that is so close that would make the whole issue of the Irish border almost irrelevant.
"But I'm not so naive to think that that's likely, so that's why we do need to have a backstop, and that backstop can't really apply to the UK as a whole because the EU is willing to make concessions, is willing to make a special case for Northern Ireland, willing to give it special treatment because it's a small place, less than two million people and because of its unique history and geography that special treatment or access to the single market unfortunately can't be on offer to the UK as a whole," said Mr Varadkar.
Mr Varadkar said he believed there would be progress over the coming weeks "because there has to be".
He added: "Time is running out. It is in absolutely nobody's interest that we end up in a no-deal scenario. Britain and Ireland would have the most to lose from that, but all of Europe would lose as well."
The Taoiseach said what he called desktop-based contingency planning for a no-deal scenario was on an east-west basis only, and not north-south.
Some of this would be placed in the public domain in the coming weeks.
What a no-deal scenario would mean for the border in Ireland was "just something we're not contemplating", he said.
"Everyone understands here in Brussels, in European capitals, in London, that that's something that's [not happening]. It's not being planned for, and we won't agree to anything which gives rise to a hard border on our island, and if we do end up in a no-deal scenario we'll have to talk to our EU partners about that.
"But bear in mind that that would only arise if the United Kingdom started to diverge in terms of regulations and customs arrangements, so they would be the ones who would be causing it."
EU demands clarity from May over Brexit trade deal
Earlier, European Union leaders demanded clarity from Mrs May on her plans for a post-Brexit trade deal.
In conclusions swiftly adopted by the leaders of the remaining 27 European Union nations at the summit, they called for "realistic and workable" Brexit proposals from Mrs May.
In their conclusions the 27 leaders welcomed progress on the legal text of the withdrawal agreement but noted that "important aspects still need to be agreed", including the territorial application of the deal "notably as regards Gibraltar" - a provision thought to have been inserted at the insistence of Spain.
The 27 leaders also expressed concern that "no substantial progress has yet been achieved on agreeing a backstop solution for Ireland/Northern Ireland".
They demanded "further clarity as well as realistic and workable proposals from the UK as regards its position on the future relationship".
And in a hint that UK red lines could be softened, the 27 leaders said "if the UK positions were to evolve" the EU "will be prepared to reconsider its offer".
In a sign that no deal was still a potential outcome, the leaders called on member states and EU institutions "to step up their work on preparedness at all levels and for all outcomes".
Brussels' chief negotiator Michel Barnier warned that time was short and suggested an extra round of negotiations on Monday to help reinvigorate the talks.
Mr Barnier said "huge and serious divergence" remained over issues relating to Ireland and Northern Ireland.
Mrs May said she hoped a new phase in the Brexit talks would be possible after the publication of the British government's White Paper calling for negotiations to speed up and intensify once the document is published.
Mrs May's participation in the European Council summit ended in the early hours of this morning after a marathon session on proposals to address the migrant crisis.
She warned EU leaders the safety of their citizens would be put at risk if a deal on security co-operation after Brexit was not reached.
Leaving the summit in Brussels, she said: "We are going to be publishing our White Paper shortly and I want to see the negotiations accelerating and intensifying thereafter."
She urged her counterparts to change the mandate given to their negotiators in order to allow UK involvement in key law enforcement initiatives allowing police and security agencies to share information.
Without a deal, she told her counterparts, "we would no longer be able to share real-time alerts for wanted persons, including serious criminals" and "our collective ability to map terrorist networks across Europe and bring those responsible to justice would be reduced".
The remaining 27 EU leaders agreed their joint position on Brexit in Mrs May's absence today.
Mr Barnier said the EU wanted an "ambitious" partnership but it had to be based "on our values and our principles".
In a light-hearted bid to ease tensions, Belgian premier Charles Michel surprised Mrs May with a gift of a Belgian football shirt.
A World Cup clash between Belgium and England coincided with the summit last night. Belgium won 1-0.