Boeing's Starliner spacecraft will not achieve its mission objective of docking with the International Space Station, NASA has said.
The announcement deals a blow to the agency's plans to end US dependence on Russian rockets for astronaut taxi rides.
Officials said the autonomously flown capsule experienced a glitch involving its onboard clock that led it to burn too much propellant, forcing an early return to Earth on Sunday morning.
"We have made a final decision - Starliner will not dock with the @Space_Station and will return to White Sands on Sunday," tweeted NASA Administrator Jim Bridenstine.
The failure of the mission, a final dress rehearsal before a crewed flight, will be seen as especially stinging for Boeing, which is facing a safety crisis over its grounded 737 MAX planes.
After the Space Shuttle program was shuttered in 2011, NASA awarded Boeing and Elon Musk's SpaceX contracts worth billions of dollars to provide transport for US astronauts.
Both companies are two years behind schedule, but SpaceX carried out a successful autonomous rendezvous and docking with the ISS in March.
Starliner, which was fixed to a giant Atlas V rocket, took off before sunrise at 6.36am local time (11.36am Irish time) from Cape Canaveral, separating 15 minutes later.
Around 90 minutes after lift-off, Boeing announced on Twitter it had an "off-nominal insertion," indicating the procedure to even out its orbit had not gone as planned, and a live stream was cut shortly after.
Mr Bridenstine told reporters the Starliner's on board clock was out of sync, "and that anomaly resulted in the vehicle believing that the time was different than it actually was."
Assuming it was at a different stage of its flight, Starliner burned more fuel than it should have, forcing NASA and Boeing to call off the rendezvous with the ISS.
Mission control attempted to manually override the problem from the ground, but they were unable to establish a connection in time because of a satellite communication link failure.
Starliner will instead return to Earth, landing at NASA's White Sands facility in the New Mexico desert on Sunday morning around 7.30 am local time.
Under former president Barack Obama, NASA opted for a shift in how it operates: instead of owning the hardware, it decided to hire private companies to take over the role, awarding Boeing and SpaceX billions of dollars to develop "Made in the USA" solutions.
The developments are independent of the Artemis program to return to the Moon by 2024, which will use a spaceship built for longer journeys, Lockheed Martin's Orion.