The Republican president has advocated prison reform and said he was open to changes in sentencing guidelines, an issue that divides the Republican Party between "law and order" hardliners and moderates.
"There are people in jail for really long terms," Trump said in an interview with Fox News. "We do need reform. And that doesn't mean easy. We're going to make certain categories tougher when it comes to drug dealing and other things.
"It's very unfair to African-Americans it's very unfair to everybody. And it's also very costly."
He said he would overrule his attorney general, Jeff Sessions, who has baulked at efforts to reduce sentences served by convicts and is unlikely to support new ones.
"Well if he doesn't then he gets overruled by me, because I make the decisions, he doesn't," Trump said.
Trump planned to have lunch with West, one of his most famous celebrity supporters, and Brown, a prison reform advocate.
In June, Trump commuted the life sentence of a woman for a first-time drug offence whose cause was taken up by celebrity Kanye West's wife, Kim Kardashian West. Alice Marie Johnson, a 63-year-old from Tennessee, had served more than 20 years in prison on drug conspiracy and money laundering charges.
Trump claims Kanye West as a friend and a "different kind of a guy" whose big following in the African-American community helped boost his approval ratings. Asked if he would consider campaigning with the rapper, Trump said, "I could see it. I could see it."
(With inputs from agencies.)