He was speaking at an interactive session hosted by the All India Professionals Congress, an organisation affiliated to the Congress party, here.
Asked about the recent turmoil within the CBI, where the two senior most officials have levelled allegations against each other, Chelameswar said that "nobody issued (defined) a legal structure of the organisation" in the last seventy years.
"It is not a constitutional body. It is not a statutory body. This setup almost has tantalising powers," he said.
The CBI was set up under the Delhi Special Police Establishment Act.
Every time something happens at the state-level with political implications, people want the case to be referred to the CBI, forgetting that the agency also consists of human beings and they can make mistakes, he said.
"Every political party could work towards creating a proper statutory framework for the organisation," the retired judge added.
Earlier this year, Chelameswar was among the four senior judges of the Supreme Court who went public with their grievances about then Chief Justice of India Dipak Misra's style of functioning.
Asked about that unprecedented press conference, Chelameswar said going public was the only option available to them.
The press conference was the culmination of a series of events that started in November 2017, and the decision to go public was taken only after a letter written to then CJI did not yield results, he said.
"There was a lot of criticism over our going public. If there was any possibility of issues getting solved internally, why would any of us venture out," he said.
Asked about the possibility of the Supreme Court coming in the ambit of the Right to Information Act, he said judges already give reasoning for the legal decisions taken by them but there could be scope for bringing administrative decisions within the RTI purview.
Asked to comment on absconding businessman Vijay Mallya's lawyers contending in the UK court while opposing the request for his extradition that conditions in Indian jails are bad, he said while Indian jails do not boast of "five-star facilities", certain prisoners do get five-star treatment.
He also said that he would not be joining any political party or contest polls.
"Some may think political appointments (for retired judges) are not right. But what has the civil society do about it? If the civil society thought of it as not right, they should have protested. That may not have stopped it (the appointments) but would have been a deterrent factor," he said.
(With inputs from agencies.)