Media reports claim the woman is in her fifties and the clinic's nurses had told her she was old and they could not assist her.
"On her way, just outside the CHC, she realised that she would not make it to the hospital before the baby was born. She, unfortunately, gave birth at the gate while trying to walk back into the facility," said the Gauteng Department of Health.
A team comprising quality assurance and labour relations officials are expected to report back within a month, with recommendations on the course of action.
District Chief Director Mothomone Pitsi said he is saddened by the unfortunate incident.
"It is unacceptable that the patient was not examined and let to go in her condition. Even in circumstances where patients are identified as high risk and require care at the higher-level facilities – they are not just sent away," said Pitsi.
Pitsi further expressed regret with regard to the incident and conveyed apologies to the patient, family and the public.
(With Inputs from South African Government Press Release)