Cleveland police have charged Ariel Castro with four counts of kidnapping and three counts of rape after three missing women were found in his home.
However, police said no charges have been filed against Mr Castro's brothers Onil and Pedro, who were also arrested on Monday.
Cleveland Prosecutor Victor Perez said there is no evidence they had any part in the crime.
Amanda Berry, 27, Gina DeJesus, 23, Michelle Knight, 32, and Ms Berry's six-year-old daughter were rescued on Monday.
The women had been held in the house since they disappeared between 2002 and 2004.
The kidnapping charges relate to the three women and Ms Berry's daughter, while the rape charges are for alleged attacks on the three women.
Police said the only opportunity for the women to escape came on Monday.
Ms Berry and Ms DeJesus had family homecomings today, but neither spoke to the media as they were hustled into their family homes.
Ms Knight is in a Cleveland hospital, where a spokeswoman said she was in good condition.
Detectives also said ropes and chains were among evidence collected inside the house.
However, no human remains were found in the house.
Mr Perez said Mr Castro, 52, would be arraigned on Thursday morning.
Initial public euphoria over the rescue of the women has given way to questions of how their imprisonment in a house on a residential street went undetected for so long.
Several neighbours said they had called police to report suspicious activity at the house in a dilapidated neighbourhood on Cleveland's West Side.
However, police denied those calls from neighbours were made.
"We have no record of those calls coming in over the last ten years," Cleveland Police Chief Michael McGrath said on NBC's Today show.
Mr McGrath said he was confident police did not miss opportunities to find the missing women.
"Absolutely, there's no question about it," he said.