By the 13th century Huntingdon was only very slightly smaller that today. At this time there were sixteen parishes with churches in its confines plus six religious houses around the Borough of Huntingdonshire with St Mary's priory owning nine of the sixteen parishes.
There was Hinchingbrooke Priory in the west, St Margaret's in the north (on the site of the modern Spittal's Way), a leper hospital of St Giles, and the Austin Friars, whose lands and buildings ended up with the Cromwells. Oliver Cromwell was born within these refurbished buildings. In addition some 17 priories and abbeys had call upon lands and titles within the parishes of Huntingdon.

However the course of the River Ouse to Kings Lynn was being restricted, with mill-pools and sluices diverting the water and narrowing the waterway. A large mill remains, only a few miles away at Houghton, now owned by the National Trust, and open to the public. The bridge at St Ives, plus the granting of a fair also lead to Huntingdon's decline, with loss of revenues from tolls having an effect on the town's dwindling finances.

The final blow was the coming of the Black Death, which seemed to hit the town especially hard. By 1363 one quarter of Huntingdon was uninhabited, the remaining residents scarcely scraping a living, and facing heavy taxation. In 1364 three parish churches were reported derelict and eight others which were not allocated incumbents from the 14th century shared the same fate.

Some hoped a new charter in 1363 would help, by giving the town authorities (burgesses) the right to exact payment from strangers renting houses, or hiring storage facilities. Later, the right to confiscate the belongings of all outlaws and felons found within the borough was put to extensive use during the Peasants Rising of 1381.But despite this, during the 15th and 16th centuries, the burden of taxes took its toll on Huntingdon and burgesses fled rather than take office.