The echo of hobnail boots coming down the road was the signal that the "home babies" of Tuam were heading to school.
The illegitimate children of so-called "unmarried mothers" were, he said, considered "the children of sin".
They were "the devil's children, they weren't encouraged to mix".
The children are believed to have died from natural causes, but the search for their remains has raised questions about the living conditions and practices within such Church-run institutions.
Mother and baby homes operated across Ireland from 1925 to 1961 and housed thousands of women who had become pregnant outside marriage.
The home in Tuam, County Galway, was run by the Bon Secours order of nuns and a child died there nearly every two weeks between the mid-1920s and 1960s.
The mass grave has been labelled "a chamber of horrors" and a "social and cultural sepulchre" by Taoiseach Enda Kenny.
Irish police are liaising with the local coroner over the find.
Mr Dwyer, a former secondary school principal, told Radio Ulster's Sunday Sequence that the children from the home were kept apart from the other children at school.
"They were regarded as the children of sin," he said. "They were only little children, aged four to seven years of age, it was a horrible way to treat them."
They grew up, he said, behind the high walls of the mother and baby home - set up by the Church for women who became pregnant outside marriage and their illegitimate children.
"The walls were 8 ft to 12 ft high and the kids were in behind. The only opportunity to meet was at school. But they were segregated from us," Mr Dwyer said.
"They came in a line down the road and they wore hobnail boots. If you heard them, you knew you were late for school.
"I remember boys from the home being strapped for being late, even though they had no control at all over the time they would arrive," he said.
As a child, he felt the treatment of the "home babies" was wrong.
"We weren't allowed to mingle with them. If you were misbehaving, the threat was that you would be put with the home babies and that was frightening as you wouldn't see your mammy. There was also the stigma," he said.
But one girl stands out for him and he is on a mission to find her.
"I was being bullied by one of the town children and one of the town babies stepped forward and said 'You better stop bullying him, or I will deal with you'.
"She became my protector," he said.
Now, he would like to make contact with her and find out what happened to her in life.
The Catholic Church had enormous power in those days, particularly in rural areas, Mr Dwyer said.
"They had immense control. When mothers became pregnant, they were collected and taken away."
But within the community, there was a tolerance of young women getting pregnant. He remembers one case where a grandmother brought up her grandchild as her own. It was tacitly accepted in the community.
"The Church was all powerful - very rarely did anybody question what went on. That was the society of the time.
"Society does have to look at itself.
"Our own archbishop in Tuam is looking for where it all went wrong, wondering why the unmarried mothers were stigmatised and ostracised by their own community and the Church."
But Mr Dwyer said there were no excuses for the fact that nearly 800 babies and young children were disposed of following their deaths and that there was no proper Christian burial.
The Tuam site is being excavated as part of a government-appointed inquiry into mother and baby homes.