Ibrahim Hoteit told AFP he changed his mind about the judge over fears that tensions surrounding the probe could lead to civil war.
Several political parties, including powerful Shia movement Hezbollah, have accused judge Tarek Bitar of political bias in his probe into the August 4, 2020 explosion that killed more than 210 people.
Tensions came to a boil Thursday when seven people were killed in violence following a rally by Shia protesters demanding Bitar's dismissal.
The shootout in central Beirut kept many residents trapped indoors for hours, reviving memories of Lebanon's 1975-1990 civil war.
In a complete U-turn from his previous support for the investigator, Hoteit in a video circulated Friday night called for the judge to step down.
Many social media users, as well as fellow activists, speculated that Hoteit, whose brother was killed in the port explosion, had come under political pressure.
But Hoteit told AFP he had spoken only for himself and denied having recorded the video under duress.
"The only pressure I came under were the events of Thursday and the fear of sliding into civil war," he said.
"I therefore decided to step down as spokesman" for the families.
Other victims' relatives, retaining support for Bitar, said in a statement: "This position... does not represent us at all."
William Noun, whose firefighter brother died, told AFP that he remained firm in his support and that Hoteit "was without doubt pressured".
Top politicians subpoenaed in the port case have launched various legal challenges against Bitar.
But court rulings have allowed Bitar to resume the investigation, which has been suspended multiple times.