NEW YORK (CNNMoney) — Just as the government expands its disaster loan program for businesses hurt by the drought, a new report shows the agency that distributes the loans is letting millions of dollars go unpaid.

As of late last year, taxpayers were owed $171 million for delinquent disaster loans, according to an audit by the inspector general of the Small BusinessAdministration.

The low-interest loans are meant to offer temporary help to businesses, nonprofits and homeowners.

Unlike smaller, SBA-backed loan programs, disaster loans can be issued by the agency itself, and are offered only to those in declared disaster areas.

Borrowers must agree to pay back the loans. But the SBA hasn’t followed collection rules, allowing millions of dollars to stay in limbo, the inspector general audit found.

The report was issued just before the SBA expanded its disaster loan program last week to companies and nonprofits in 1,430 counties across 32 states. Heat and arduous conditions are ruining crops across the country, and the drought is hurting small businesses that rely on spending by farmers.

The agency had $4.2 billion on hand for the program at the start of its fiscal year in October 2011, said an SBA spokesperson. Those feeling the economic pain of the drought can now access those funds. The agency also simplified the online application process last month.

Related: Severe U.S. drought slams small businesses

The SBA responded to the critical report saying it is exempt from rules requiring it to refer delinquent borrowers to the Treasury Department.

When those who borrow from the SBA’s disaster loan program fall behind, they are legally allowed to work out a deal with the agency. If they still don’tpay, the SBA can garnish a person’s wages or liquidate their property.

In the worst case scenario, in which nothing is paid for 180 days, the SBA must pass along the responsibility of collecting to the Treasury.

It hasn’t done that, and overdue debt has long gone ignored.

“Management is not acknowledging the existence of the problem,” the inspector general’s office said.

The auditor found that the SBA hadn’t moved forward on 1,553 bad loans worth $171 million. The SBA had identified nearly half of those as having collateral and being ready for foreclosure, but it hadn’t yet started moving on those either.

“When the SBA delays foreclosure, it could prevent the government from ultimately recovering… defaulted debts,” the report noted.

Read the full article on CNN Money.