Hopes for a deal among major producers, including several from the Organization of the Petroleum Exporting Countries and Russia, were a main driver in a rally that lifted U.S. crude prices more than 50% from their February lows. U.S. crude settled at $41.50 a barrel on Friday.

Now, much of those gains could be eroded in a market that has already endured a turbulent year, analysts say.

U.S. crude was recently trading 4.41% lower at $38.58 a barrel and Brent down 3.94% at $41.39 a barrel.

"This is an extremely bearish scenario,” said Abhishek Deshpande, oil analyst at Natixis. "Prices could touch $30 a barrel within days.”

Steep falls in crude could also weigh on equity markets more generally. Stocks have often moved alongside oil this year. Bank shares, for instance, many of which have large energy portfolios, have been pressured by the declines in oil.

At the meeting in Doha, oil producers that supply nearly half of the world’s output failed to reach a deal to curb their production after Saudi Arabia appeared to walk away from any agreement that didn’t include geopolitical rival Iran.

Iran has ruled out capping its own production. It had only recently returned to international oil markets after sanctions were lifted in January, and the country wants to regain lost market share.

Mr. Deshpande said that most of the recent rally was supported by the idea that some form of a deal could happen. The failed talks will "seriously hurt investor confidence,” he said.

Hedge funds and other money managers expanded their net-long position in U.S. crude—or bets that prices will rise—by 11% in the week ended Tuesday, Commodity Futures Trading Commission data showed Friday. If investors now start unwinding these bullish positions, this could add pressure to prices, analysts said.

"You’re going to see an exit of long positions,” said Jason Schenker, president of Prestige Economics in Austin, Texas, adding that prices could revisit the 13-year lows reached in the first quarter. "The assumption that there was going to be an actual freeze on supply was overestimated.”

Still, other market watchers said that prices are unlikely to fall back to those previous lows because U.S. production is falling. Concerns about weaker demand in China may also be ebbing.

"You’ve got to be more confident in the market today than you were three months ago,” said David Zusman, chief investment officer at Talara Capital Management, which manages $500 million. "I don’t think actions of OPEC will be required to balance this market anyhow. You’ve got a combination of improved demand as well as declining supplies.”

Steven Kopits, president of consulting firm Princeton Energy Advisors, added, "the view that the market will balance in the third quarter has gained some traction.”

Even those investors who had been skeptical that a deal would make a huge difference to production—not least because big producers would be freezing their crude output at current high levels—believed agreement in Doha would have sent a positive signal to the market.

A deal would have indicated that producers were willing to tackle an oversupply that has pressured prices for almost two years now, they said.