Danette O'Neal, PhD

The Federal Housing Finance Agency said last month it was considering a reduction in the loan limits, which are set at $417,000 for most of the U.S. but rise to as high as $625,500 in high-cost housing markets such as Los Angeles and New York.

Typically, loan limits were indexed to home prices, and they would rise annually depending on the median U.S. home price. But loan limits never fell once home prices did.

In a letter to the FHFA sent Tuesday, the National Association of Realtors challenged the regulator’s legal authority for such a move, arguing that Congress had been explicit about its intentions to prevent loan limits from declining.

The FHFA has said that it has broad powers as the conservator overseeing the government-controlled mortgage companies that could allow it to issue special directives limiting the firms’ ability to purchase loans above a reduced loan limit. The NAR pre-emptively challenged that legal basis in its letter.

“If you had the authority to ignore the prohibition against reducing loan limits, what would prevent you from making other fundamental changes?” said Gary Thomas, the NAR’s president.

A spokeswoman for the FHFA didn’t immediately reply to a request for comment.

The NAR also argued that reducing the loan limits would be bad policy. While rates on jumbo mortgages—those too large for government backing—have fallen close to parity with rates on conforming loans that are eligible for backing by Fannie and Freddie, jumbo mortgages typically require larger down payments and higher credit scores.

“An arbitrary reduction in existing limits, in the hope it will encourage more private sector lending, is a social policy experiment that risks dampening or reversing the ongoing recovery in the housing market and the economy as a whole,” wrote Mr. Thomas.

The NAR letter doesn’t stop there. It also argues that Fannie and Freddie—at the FHFA’s direction—shouldn’t continue to increase the guarantee fees that the firms charge lenders, which are often passed onto borrowers in the form of higher mortgage rates.