The pledges that individual countries are offering ahead of the Paris climate summit in December are too entrenched in self interest instead of being focussed on a common goal.
The researchers say the science of cooperation is being ignored.
Instead, they say the negotiations should focus on a common commitment on the global price of carbon.
This means countries would agree on a uniform charge for carbon pollution, a scheme that would encourage polluters to reduce their emissions.
The comments from researchers at the University of Cambridge, UK, University of Maryland, US, and University of Cologne, in Germany, are published in the journal Nature.
'I will, if you will'
Ahead of December's United Nations climate meeting, individual countries have submitted their plans for cutting greenhouse gas emissions. These are called Intended Nationally Determined Contributions - or INDCs.
However, the researchers say that this approach will not work.
Prof David MacKay, from the University of Cambridge, who was former chief scientific advisor to Britain's Department of Energy and Climate Change (DECC), said: "The science of cooperation predicts that if all you are doing is naming individual contributions - offers that aren't coupled to each other - then you'll end up with a relatively poor outcome.
"We have the history of the Kyoto agreement as an example of this. Initially, the approach was to find a common commitment, but eventually it descended into a patchwork of individual commitments... and that led to very weak commitments and several countries leaving the process."
The Paris negotiations, he warned, were heading in the same direction.
Instead, the researchers say, a reciprocal approach could transform the meeting.
"If you make a treaty that is based on reciprocity, so 'I will, if you will' and 'I won't, if you won't', then you can end up in a very different position," explained Prof MacKay.
"If people make a common commitment that they will match what others do, then it becomes in your self interest to advocate a high level of action because it will apply not only to you but also to others."
The scientists believe that setting a common price for carbon, which could be implemented through carbon tax or emissions trading schemes, could work.
Prof MacKay said: "This is a price that could be negotiated and agreed, and would apply to all countries."
The researchers admit that with the Paris climate conference just weeks away and the fact that global carbon pricing is not already on the table, their idea is unlikely to have much influence.
However, they say the science of cooperation should be taken into account for future negotiations.
Commenting on the paper, Bob Ward from the Grantham Research Institute on Climate Change and the Environment, said:
"This is a thoughtful contribution but rather too pessimistic about the United Nations climate change negotiations.
"The authors are right that a global price on carbon is necessary, although it would be, on its own, insufficient to generate the pace and scale of action required."