During the early 19th century, much of Salem's wealth was linked to maritime trade thus making the Custom House a critically important building in the community. Architecturally the building is a fine example of the Federal Style.

This is an early Federal building representative of governmental activity during a period when the Merchant Marine was growing rapidly. It was built for a customs office and a postal office.

Completed in 1836, the U.S. Custom House in New Bedford is the oldest continuously operating Custom House in the Nation. It is also the largest, most elaborate and arguably the finest of the series of four granite Greek Revival customhouses in New England designed by Robert Mills between 1834 and 1836. The only other customhouse designed by Robert Mills in Massachusetts is in Newburyport. Mills, the first architect of international reputation both born and trained in America, is also known for designing important buildings and structures in Washington, D.C. such as the U.S. Treasury, the General Post Office, the Patent Office and the Washington Monument.

The 39th Congress (1866) passed a bill for the purchase of a site and erection of a building at St. Paul, for a custom house, Post Office, federal coort and other necessary government offices.

The land was purchased in 1867, building started in 1868, and completed in 1875.