Northern Ireland could be given joint EU and UK status and a "buffer zone" on its border with the Republic, under new plans being drawn up by UK Brexit Secretary David Davis.
The scheme is said to be under consideration as a potential way of breaking the deadlock over future customs arrangements ahead of a crunch summit of European Union leaders later this month.
But the DUP has blasted the ideas as "contradictory and "half-cooked".
British Prime Minister Theresa May has divided ministers into two working groups to try to find a means of fulfilling her twin pledges to take the UK out of the European customs union while keeping the Irish border open.
The two options currently on the table - a customs partnership which would see the UK collect tariffs on behalf of the EU and a "maximum facilitation" scheme using technology to avoid border checks - have split the British cabinet down the middle.
Brussels' rejection of both schemes has been underlined by chief Brexit negotiator Michel Barnier, who said neither was "operational or acceptable".
In a film for Vice News, Mr Barnier said he had "no certainty" about the nature of the UK's future relationship with the EU.
"I can see the difficulty and intensity of this debate," he said. "We are waiting for the British to have clear positions and choices."
He told a private meeting of the European Parliament's Brexit Steering Group: "They have two proposals which are being debated with British ministers. Neither of those proposals are operational or acceptable to us."
Mr Davis's alternative "Max Fac 2" solution would scrap the technological solutions previously favoured by Mr Davis as a way of recording movements across the border without the need for checkpoints.
These were dismissed as unworkable by EU officials and raised concerns from police about possible sectarian attacks on infrastructure such as number plate recognition cameras.
Instead, Mr Davis is now considering a new "double-hatted" solution based on the model in place in Liechtenstein, which would allow the six counties to operate both UK and EU regulations at the same time.
A 16km (10 mile) wide "special economic zone" would be created along the border, within which local traders could operate under the South's trade rules.
A British government source acknowledged it would be a challenge to secure backing for the plan from the DUP, which props up Mrs May's government at Westminster and has made clear that it does not want Northern Ireland treated differently from the rest of the UK.
Border ideas are at best contradictory, says DUP's Wilson
The DUP blasted the ideas as "contradictory and "half-cooked".
East Antrim MP Sammy Wilson said Mrs May's administration had not discussed any of the latest reported proposals with the DUP, saying they were "at best contradictory".
He said the "convoluted" arrangements were only being discussed because the British government had failed to "make it clear to the EU that regardless of (Michel) Barnier and EU negotiators' attempts to keep us in the customs union and the single market, we are leaving.
He added: "Instead of moving from one set of half-cooked ideas to the other, it is now time for the government to put down its foot and make it clear to EU negotiators that the prime minister stands by her commitment that no deal is better than a bad deal, and if they want to avoid the consequences then they need to stop dismissing the perfectly feasible ideas that were put forward in August of last year."
Earlier, Sinn Féin accused Mr Davis of trying to "hide a hard border in a buffer zone".
MEP Martina Anderson said: "Once again this shows the lack of knowledge of border areas and the concerns they face - David Davis obviously didn't learn much on his flying visits."
Mr Davis's Department for Exiting the EU did not deny that the proposal was under consideration.
A spokesman said only: "We have set out two viable future customs arrangements with the EU and work is ongoing to refine these.
"Both of these would deliver on our commitments to ensure UK-EU trade is as frictionless as possible, avoid a hard border between Northern Ireland and Ireland, preserve the integrity of the UK's internal market and enable us to establish an independent international trade policy."
A Downing Street spokesman said: "The Prime Minister has been absolutely clear that we cannot and will not accept a customs border down the Irish Sea, and that we will preserve the integrity of the UK's common market.
"Work is ongoing on customs plans that will achieve this, as well as ensuring we can strike trade deals around the world, that trade remains as frictionless as possible, and that there is no hard border between Northern Ireland and Ireland."
