Here’s mayor and Harvard president Josiah Quincy’s earliest memory, from 1775, when he was three. It was early in the siege of 1775-76, when his mother, Abigail (Phillips) Quincy got permission to leave Boston with him:
It was my lot to be with my mother and her two sisters in that carriage. The following fact probably impressed my childish fancy, and was the occasion of my remembering it. The small-pox was then in Boston, and was at that day the terror of the country. At the line which separates Boston and Roxbury there were troops stationed, and a sentry-box on the east side of the street erected. At this point the carriage was stopped, all its inmates made to descend and enter the sentry-box successively. On each side of the box was a small platform, round which each of the inmates was compelled to walk, and remain until our clothes were thoroughly fumigated with the fumes of brimstone cast upon a body of coals in the centre of the box. This operation was required to prevent infection.And I’m sure it worked great.
Quincy also told this story at a meeting of the Massachusetts Historical Society on 10 Dec 1857.
Since that was Quincy’s earliest memory, and his namesake father died in the spring of 1775 after months in Britain, that means Quincy had no memories of his father. Only memorabilia.