Northern Ireland's First Minister Peter Robinson has said it would be wrong to proceed with the Maze Peace centre in the absence of a broad consensus about how it will operate.
In a letter to Democratic Unionist Party MPs and assembly members, Mr Robinson said there must be a change of attitude by Sinn Féin, especially towards victims of IRA violence.
The Maze housed paramilitary prisoners from the 1970s until 2000 and critics of the new development fear it could become a shrine to terrorism.
The prison hospital where ten republican inmates, including Bobby Sands, died while on hunger strike in 1981 is among parts of the jail that were retained when the site was cleared for redevelopment.
A watchtower and one of the H Block cells were also kept.
Mr Robinson cited a number of events in recent months, including the removal of the Union flag from Belfast City Hall on all but designated days and the naming of a park after a dead IRA man, as among the reasons to veto the plans.
He said his party had consulted widely on the issue.
Mr Robinson added: "The events of recent months and in particular the insensitive attitude displayed by Sinn Féin towards the innocent victims of IRA terrorism has seriously damaged community relations and set us all back in terms of promoting genuine reconciliation and building a shared future.
"For the centre to be successful in promoting peace and reconciliation there must be a broad consensus about how it will operate. We have consulted widely and it is clear that the necessary wide-ranging consensus does not exist at present.
“It is my view that it would be wrong to proceed in the absence of a much broader consensus."
Sinn Féin MLA for Foyle Raymond McCartney described Mr Robinson's comments as a "hysterical reaction" to unionist paranoia "whipped up by rejectionists like Willie Frazer and Jim Allister".
Mr McCartney said the party has been involved in ongoing discussions with the DUP on the issue for some time.
He said people will find it strange that the DUP is turning its back on the project and it raises "very serious questions about the commitment of elements of the DUP to conflict resolution and peace building".
Planning permission for the peace centre, close to the retained structures, was granted in April and sparked a wave of protests led by the Traditional Unionist Voice and Ulster Unionist Party.
A number of victims' groups and organisations representing former security force personnel and their families also opposed the plans and called for the watchtower and H-block cells to be demolished.
The recently developed Titanic Quarter in Belfast was suggested as an alternative site.
Jeffrey Donaldson, MP for Lagan Valley in which the former jail is situated, denied the DUP had done a U-turn on the Maze.
He said: "The DUP had sought to find consensus on how the peace centre would operate.
"Clearly that has not been possible given the recent attitude of Sinn Féin towards the innocent victims of IRA terrorism which has seriously damaged community relations and set us all back in the building of reconciliation."
The DUP claimed it was still committed to participating in talks led by US diplomat Dr Richard Haas in the autumn, which aim to tackle contentious issues such as parades, flags and dealing with the past.