The US state of Texas overstepped its authority when it removed some of 460 children from a polygamist compound last month, an appeals court has ruled.

The ruling is the latest twist in an unfolding saga that has riveted Texas with lurid tales of adolescent brides, teen pregnancies and a secretive religious sect following its faith in the state.

Texas Child Protective Services and heavily armed police units last month raided the isolated compound in west Texas and removed the children in response to allegations of abuse.

But the appeals court said that the state had not proven that the children were in immediate physical danger, and therefore were improperly separated from their parents.

While the opinion covers the children of only 48 mothers, a lawyer told reporters that the ruling will likely apply to all of the children, most of whom are in foster homes across the state.

The appeals court opinion gives a lower court, which initially approved the state's actions, 10 days to act.

The compound is run by followers of jailed polygamist leader Warren Jeffs. They belong to a renegade Mormon sect known as the Fundamentalist Church of Jesus Christ of Latter-Day Saints.

The mainstream Church of Jesus Christ of Latter-day Saints renounced polygamy over a century ago and is at pains to distance itself from splinter groups such as the FLDS that continue to practice plural marriage.

FLDS men typically marry one legal wife while those who follow are ‘spiritual wives.’ This makes those women single in the eyes of the state that can entitle them and their children to various welfare benefits.

FLDS communities usually keep to themselves and lead austere lifestyles. The girls and women wear 19th century-style pioneer dresses and are taught from an early age to obey men.