Former 100m and 200m world champion Gay was suspended for a year after testing positive for a banned anabolic steroid.
The 32-year-old returned his London 2012 medal when his suspension was announced in May 2014.
Now the International Olympic Committee has told US Olympic bosses that the whole team must return their medals, BBC reported.
The IOC wrote to the US Olympic Committee on Wednesday, telling them to collect the medals from Trell Kimmons, Justin Gatlin, Ryan Bailey, Jeffery Demps and Darvis Patton.
"We will begin efforts to have the medals returned and support all measures to protect clean athletes," said a US Olympic Committee statement.
Kimmons and Bailey ran with Gay in the final, as did Gatlin, who has served two doping bans, including a four-year suspension between 2006-2010.
Gay's results from 15 July 2012 - the date he first used a product containing a banned substance - were annulled.
His one-year suspension was backdated to 23 June 2013, the date he tested positive at the US World Championship trials.
The American team finished second in London's Olympic Stadium behind Jamaica, who were anchored by Usain Bolt on their way to a new world record.
Gay and his team-mates set a new US record with their time of 37.04 seconds.
If the medals are reallocated, Trinidad and Tobago - who finished third in 38.12 seconds - would be awarded silver. Fourth-placed France would take the bronze instead.
Great Britain were disqualified in their semi-final as the final handover between Daniel Talbot and Adam Gemili was not completed in time.