Kenya got only a small slice of the disputed tract of sea off the East African coast in the ruling by the International Court of Justice (ICJ).
Kenya's President Uhuru Kenyatta responded strongly, saying his government "rejects in totality and does not recognise the findings in the decision".
With Kenya refusing to recognise the "biased" court's authority, all eyes will be on what Nairobi does next in one of the world's most troubled regions.
Somalia urged Kenya to "respect the international rule of law" following the verdict by the court based in The Hague.
Judges unanimously ruled there was "no agreed maritime boundary" in force and drew a new border close to the one claimed by Somalia.
Chief judge Joan Donoghue, reading a summary of the judgment, said the "court is thus satisfied that the adjusted line... achieves an equitable solution".
Somalia dragged Kenya to the court in 2014 after years of efforts to resolve a dispute over the 100,000 square-kilometre (38,000 square-mile) tract failed.
The ICJ's judgment is final and cannot be appealed, but the court, set up after World War II to rule in disputes between UN states, has no means of enforcing its rulings.
States can however go to the UN Security Council if another country fails to obey a ruling.
In a televised speech, Somalia's President Mohamed Abdullahi Mohamed, who is widely known as Farmajo, said Nairobi should see the ruling as an "opportunity to strengthen the relationship of the two countries".
"The verdict was a fair indication of the transparency of the International Court of Justice," said Farmajo, adding that he had faced "political, diplomatic, security and economic pressure by the Kenyan leadership."
But Kenyatta said that the ruling amounted to "a zero-sum game, which will strain the relations between the two countries".
"It will also reverse the social, political and economic gains; and potentially aggravate the peace and security situation in the fragile Horn of Africa Region," he added, reiterating Nairobi's support for a negotiated settlement instead.
At the heart of the Somalia-Kenya dispute was the direction that their joint maritime boundary should take from the point where their land frontiers meet on the coast.
Somalia insisted the boundary should follow the orientation of its land border and thus head out in a 200 nautical-mile line towards the southeast.
But Kenya said its boundary runs in a straight line due east -- a delineation that would have given it a big triangular slice of the sea.
Nairobi says it has exercised sovereignty over the area since 1979.
The contested area is believed to contain rich gas and oil deposits, and also has important fishing rights. Kenya has already granted exploration permits to Italian energy giant ENI but Somalia is contesting the move.
The ICJ on Tuesday threw out Kenya's argument that Somalia had "acquiesced" to its claims by not protesting more strongly. It also rejected the line Nairobi wanted to draw from the coast, saying it would have a "severe cut-off effect" for Somalia.
But it adjusted Mogadishu's proposal slightly, saying Kenya risked having its own maritime rights sandwiched between Somalia to the north and Tanzania to the south.
Finally, the court dismissed all of Somalia's claims for damages over surveying and drilling by energy firms that Kenya had allowed.
Kenya pulled out of hearings in the Somalia case in March, then just over two weeks ago said it would not accept the court's jurisdiction due to its "obvious and inherent bias".
Legal analyst Cecily Rose of Leiden University said Kenya had relied on "long shot" arguments.
"Perhaps Kenya's decision not to participate in the oral proceedings was partly in recognition of the fact that this was a losing battle," Rose told AFP.
Monday's verdict may further sour diplomatic relations between the two countries after Kenya in 2019 recalled its ambassador and accused Somalia of selling off oil and gas blocks in the contested area.