The CST-100 Starliner astronaut capsule successfully launched earlier from Cape Canaveral in Florida, but an automated timer error prevented the spacecraft from reaching the orbit that would have put it on track to meet up with the space station.
The debut journey to the space station was a milestone test for Boeing, which is vying with SpaceX to revive NASA's human spaceflight capabilities. SpaceX's Crew Dragon capsule carried out its unmanned test flight to the space station in March.
The thwarted mission took place as Boeing, whose shares dropped about 1% in morning trading, sought an engineering and public relations win in a year punctuated by a corporate crisis over a ban on its 737 MAX jetliner following fatal crashes.
The plan was now for the capsule to head back to Earth, landing at White Sands, New Mexico on Sunday, Boeing's space chief executive Jim Chilton said.
This is the landing site that would have been used if the capsule had completed its planned week-long stay at the space station.
The craft had burned too much fuel to risk trying to land at the International Space Station at this point, NASA Administrator Jim Bridenstine said.
The spacecraft, a cone-shaped pod with seats for seven astronauts, blasted off from Cape Canaveral at 6:36 a.m. (1136 GMT) atop an Atlas V rocket from Boeing-Lockheed Martin Corp's United Launch Alliance.
Minutes after liftoff, Starliner detached from the main rocket booster, aiming for a rendezvous some 254 miles (409 km) into space on Saturday morning with the space station.
Both Boeing and NASA said the capsule was in a stable position despite not reaching its intended orbit.
"We did obviously have some challenges today. When the spacecraft separated from the launch vehicle we did not get the orbital insertion burn that we were hoping for," Bridenstine said.
He said the timer error caused the capsule to burn much of its fuel too soon, preventing it from reaching the desired orbit. NASA and Boeing made attempts to manually override the automated errors, but two satellites obstructed its communication signals, he said.
"The challenge here has to do with automation," Bridenstine said of the unmanned craft, adding that if astronauts had been on board they would have been able to override the automated system that caused the error.
Nicole Mann, one of three astronauts slated to fly on Boeing's first crewed flight test, told the news conference, "We are looking forward to flying on Starliner. We don't have any safety concerns." NASA astronaut Mike Fincke added, "Had we been on board, we could have given the flight control team more options on what to do in this situation."
SPACE RACE SETBACK
The test is one of the most daunting milestones required by NASA's Commercial Crew Program to certify the capsule for eventual human spaceflight - a long-delayed goal set back years by development hurdles from both Boeing and SpaceX.
In November, a blistering government watchdog report found Boeing received an "unnecessary" contract boost from NASA.
Boeing was awarded $4.2 billion and Elon Musk's SpaceX $2.5 billion by the U.S. space agency in 2014 to develop separate capsule systems capable of ferrying astronauts to the space station from American soil for the first time since the U.S. Space Shuttle was retired from service in 2011.
SpaceX declined to comment on the Boeing launch result on Friday.
The program, primarily meant to end America's reliance on Russia's space program for rides to the space station, had initially expected its first crewed flights on Starliner and SpaceX's Crew Dragon capsule in late 2017.
Design and safety concerns for both vehicles have led to delays.
The current target date for the first crewed flight is planned for mid-2020 for Boeing and SpaceX.
Occupying one of Starliner's astronaut seats on Friday was a mannequin named Rosie, outfitted with sensors to measure the pressure a real astronaut would endure on ascent to the space station and during hypersonic re-entry back through Earth's atmosphere.
If the mission had gone to plan, the Starliner would have stayed at the space station before beginning its return to Earth on Dec. 28.