Meanwhile, the coast guard declared it had thoroughly searched an area of 180 square nautical miles, and at 9:30 p.m. officially ended the search.
On Saturday night, a five-metre open boat was declared missing, and early Sunday morning, a fishing vessel found the first victim of the accident - a 10-year-old boy.
Around noon Sunday, the body of his 12-year-old brother was recovered.
The body of a middle-aged man, who was a friend of the boys' family, was also found around the same time
"They weren't too far apart," said Twillingate mayor and harbourmaster Gordon Noseworthy.
"All indications are that they were alive in the water, and they were holding on to each other. And then when they succumbed to their death, that's when they drifted apart."
The fourth man, who is still missing, reportedly lives in Yellowknife, and was home for a one-month visit.
Later in the evening, just before the start of a candelight vigil, Noseworthy released the names of the three confirmed dead.
Paul Froude, was the adult discovered dead, and Josh and James Guy were the 10- and 13-year-old brothers.
Noseworthy said the two boys were well known around town.
"They're different," he said. "There's no video games in their life. They're on the wharf every day of the week."
Recently, Noseworthy said, the Lion's Club advertised for volunteers to help re-shingle their roof.
"Them two little brutes showed up and they were there for the day," he said. "When they had (the project) all done, you never had to pick up any garbage, they had it all picked up."
While hope was fading, Noseworthy said he hopes the body of the fourth man will be found, to give the town and his family a sense of closure.
"It's not easy for anybody in the community to deal with," he said. "With the fourth body, if we don't recover it, it's going to leave a gaping hole in the whole situation."
"It's not easy for anybody in the community to deal with." - Gordon Noseworthy
The man will be declared "missing at sea" and the file will be turned over to the RCMP as a missing persons case.
Neil Peet, acting superintendent of search and rescue, said not finding the man has been "disheartening" for the search crews.
"We'd like to bring everyone home to the families to provide closure, and when we have an individual missing at sea, it's hard on our search and rescue resources," he said.
"It's somewhat frustrating to them."
Originally, searchers had believed that all four men were wearing life-jackets.
However, Peet said they now believe the man still missing was not wearing one. They have not been able to find the boat and believe it may have sunk,
Both boys were wearing life-jackets.
The were known to hang around down at the wharf and would often go for boat rides.
On Saturday, they went to their house to get their life-jackets and permission from their mother to make the trip.
"They got their life-jackets, their mother came down on the wharf and made sure they had their jackets on and told them to be good," he said.
"She did say, 'Now, if anything ever happens and you end up in the water, you get ashore and light the biggest kind of fire; someone will pick you up.'
"She did tell them that."