Scotland's Cabinet Secretary for Rural Economy has said Irish vessels could be boarded if they do not stop fishing in the waters around Rockall.
The Scottish government last week threatened to take action against Irish vessels that it says are fishing illegally around Rockall.
The uninhabited rocky outcrop lies around 230 nautical miles northwest of Donegal and 240 miles west of Scotland in the North Atlantic.
Fergus Ewing, who serves as Scotland's Fisheries Minister, said it was disappointing that Irish fishing activity in the area was continuing.
Speaking on RTÉ's This week, he said the Scottish government's threat of action was entirely routine and part of ongoing enforcement of fisheries legislation.
Mr Ewing said Irish vessels had never been allowed to fish in this way in the UK's territorial sea around Rockall.
He said that there was a set prescribed process that involves taking appropriate action.
Mr Ewing said: "The normal approach is to invite the captain to cease and desist. If that instruction is obeyed, then there's no need for further action.
"If it's necessary to do more, then the vessel would be boarded and action would be taken in accordance with the law."
However, the Minister for Agriculture, Food and the Marine has said he would not be asking Irish fishermen to leave waters around Rockall, despite the threat of enforcement action by the Scottish authorities.
Speaking on the same programme, Michael Creed said Ireland had never recognised UK jurisdiction over Rockall and that since Scotland passed the Island of Rockall Act in 1972, it had never sought to enforce a 12-mile exclusion zone.
He said that Ireland had a right to fish there under EU common fisheries law and that it had a quota known as Rockall Haddock Quota.
He said: "Our fishermen that are there now, and those that are planning to go there, are doing so under EU law.
"We believe they are legally entitled to be there and we won't be asking our fishermen to leave the region around Rockall."
Minister Creed said the Government would exhaust all legal options in the context of defending the interests of the State and Ireland's fishing industry.
He said he believed a sensible solution could be found and that Scotland's threat of enforcement could be withdrawn.
Mr Ewing also said it has always been Scotland's preference that the Irish Government intervened and advised Irish fishermen to cease and desist.
He said it was the expectation of fishermen in Scotland that their government would take steps to enforce the law.
When asked about Ireland's activity in the area over the last number of years, he said that it had never been authorised and that persistence did not make it legal.
Mr Ewing said the matter had been repeatedly raised at an official ministerial and diplomatic level, and that it had nothing to do with Brexit.
Meanwhile, Sinn Féin TD Pearse Doherty said the Government should deploy boats from the Sea Fisheries Protection Authority to protect Irish fishermen at Rockall if diplomatic efforts fail.
The Donegal TD said: "What Scotland is doing is unjustified and illegal. We need to escalate the diplomatic discussions.
"While those escalations need to take place, let me be clear there should be no circumstances where the British authorities are impounding Irish vessels or their equipment.
"There should be no case where the Scottish authorities are boarding our ships, impounding our vessels, and taking our nets and putting the livelihoods of our fishermen at risk.
"The Sea Fisheries Protection Authority is the police force of the sea. The idea that they would sit back and allow another jurisdiction to board our fishing fleet is not justifiable
"So if there is a complete breakdown of the diplomatic discussions then of course we need to deploy our sea fisheries protection agency and make sure that fishermen who are carrying out their work legally and within the common fisheries policy are not hindered by any other agency in doing that."