ORLANDO, Fla. – Lou Pearlman and federal authorities have finally agreed on how much the former boy band promoter swindled from banks and investors in a decades-long scam: a staggering $300 million.

That’s how much creator of the Backstreet Boys and ‘N Sync will have to repay, at a minimum, for restitution on the fraud conviction for which he’s serving a 25-year prison sentence.

U.S. District Judge G. Kendall Sharpe on Wednesday asked prosecutors and defense attorneys to amend court documents with the agreed amount.

It will be difficult for Pearlman to repay all the money while he is behind bars. Pearlman made millions in the record industry in the 1990s, but investigators have found that money and more seemingly gone with the collapse of his Ponzi scheme.

He’s been allowed to manage – at arm’s length – the few remaining music acts he still has. He could also offer wages from whatever job he gets in federal prison, ranging from 12 cents an hour to $1.15 an hour for top-scale factory work.

Attorneys from both sides, the FBI and FDIC determined Pearlman took $195 million from more than 1,000 people in an alleged savings program promising 6 percent to 10 percent returns, and $126.7 million in bogus loans from federally insured banks. Another $70 million was invested by people who thought they were buying shares in companies owned by Pearlman that mostly had no assets. About $95 million was returned to investors over the years, documents show.

But Pearlman’s restitution could go up. Sharpe delayed judgment on prosecutors’ request to tack on $124 million in interest payments to victims, saying he wanted to see Pearlman return some of the principal first.

“If they had not provided their money to Mr. Pearlman, they would have received interest or some return on their investment,” Assistant U.S. Attorney Roger Handberg argued in court.

Pearlman’s attorney, former public defender Fletcher Peacock, said adding interest would only dissuade Pearlman from repaying any of his debt. Peacock also said it was unfair because the money was never invested.

“This is not a case of where Mr. Pearlman accrued an amount of interest and objected to giving that back to people,” Peacock said. “This interest was fictional.”

In May, Sharpe said he would shave a month off Pearlman’s sentence for each $1 million he returned. So far, Handberg said, no additional money has been recovered.

A federally appointed trustee has found few assets to compensate victims, some of whom lost their life savings.

The judge has ordered that individual investors be repaid first, then banks. He wanted them punished for poorly judging Pearlman worthy of multimillion dollar loans, many secured with the same collateral.

While reserving judgment on interest, Sharpe said it was unfair to charge Pearlman the high rates he promised investors because it would reward their poor decisions.

“Since the time of the sentencing all you’ve gotten from the defendant is the smirk on his face,” Sharpe told prosecutors. “So let’s try to get some money first.”

Sharpe remanded Pearlman to the Federal Bureau of Prisons, which will transfer him to an undetermined facility. Pearlman had been at the Orange County jail, a few miles from his opulent former offices.