Gunmen have ambushed and killed five Malaysian policemen as fears mounted that armed intruders from the southern Philippines had slipped into at least three coastal districts on Borneo island.

Two of the attackers were also fatally shot last night, escalating tensions in eastern Sabah state.

Malaysia's biggest security crisis in recent years began after about 200 members of a Philippine Muslim royal clan occupied a village last month to claim the territory as their own.

Security forces clashed with the clan members in the coastal area of Lahad Datu on Friday, leaving 12 Filipinos and two Malaysian police commandos dead.

The remaining clan members have refused to budge, while concerns have grown that other groups from the Philippines' restive southern provinces might enter Sabah, which shares a long and porous sea border with the Philippines that's difficult to patrol.

A police team was attacked late Saturday while inspecting a settlement in Semporna town, more than 150km from Lahad Datu, said national police chief Ismail Omar.

Authorities were searching the area for more of the assailants.

Police are also investigating sightings of armed foreigners in military-style clothing in a third Sabah seaside district nearby, Ismail said.

It was not clear whether the groups in the three areas had links to each other.

The Filipinos who landed in Lahad Datu on 9 February say ownership documents from the late 1800s prove the territory is theirs.

They have rejected repeated calls from both the Malaysian and Philippine governments to leave Sabah, a short boat ride from the southern Philippines.

Malaysian Prime Minister Najib Razak said the government would offer "no compromise - either they surrender or face the consequences if they refuse."

Police dropped leaflets by helicopter over the occupied village Saturday telling the Filipinos to give up, while the navy bolstered patrols in waters between Malaysia and the Philippines.

Three of the intruders tried to escape yesterday and were caught, Ismail said, without elaborating.

The standoff has raised the Sabah territorial issue, a thorn in Philippine-Malaysian relations for decades, to a national security concern for both countries.

The crisis erupted during a delicate time in peace talks brokered by Malaysia between Manila and the Moro Islamic Liberation Front, the main Muslim rebel group in the southern Philippines.