Making Hay with the Hancock Papers

Dr. Jeffrey M. Griffith’s John Hancock Papers website uses generative A.I. in a way I haven’t seen elsewhere: to answer questions about Hancock and his career based on a database of over 4,000 documents.

It looks like it’s not possible to search for particular words or phrases in the Hancock letters, as one can do at Founders Online. That might come later, after publication of the volumes through the Colonial Society of Massachusetts. Or that approach might reflect my traditional thinking.

Instead, the “search” function is designed around topics, as interpreted through large language model A.I. software analyzing the transcribed letters. The website explains:

Dr. JMG’s AI seeks to find connections between letters, individuals, events, and subjects. The technology, however, requires extensive training – and will become more comprehensive as additional letters are uploaded and questions are posed to the platform. As a Private AI system, all responses to prompts are entirely generated from the content of the uploaded letters on this website and are not impacted by external sources.

The responses point the user to the primary source material. Any interpretation provided by Dr. JMG's AI response should not be considered authoritative, but rather as a step for the user to interpret and interact with the primary source material.

The queries and answers appear on the Community Bulletin Board, an archive going back to spring 2025. Early on we can see the program responding then that it doesn’t yet have information to answer questions, later (presumably after more documents have been entered into the archive) providing answers. Presumably, we see evidence of the software being “trained.”

One example caught my eye. On 25 Aug 2025, the system responded to the query “Is rain mentioned?” with this text:

Yes, rain is mentioned in the context provided. Specifically, it is noted: "It has been quite otherwise here, I have not known aHotermore Wet Summer, & We have a prospect of a great Quantity of Hay." This indicates that the summer was particularly wet, implying significant rainfall.

In a keyword search of the letters’ text, looking for “rain” wouldn’t have caught the phrase “Wet Summer.” But, depending on its sensitivity, it could have flagged words like “constraint.” For someone seeking meteorological commentary, therefore, that type of search would produce both false negatives and false positives.

A search method that could interpret “rain” as more than a series of four characters, extend that term to find “Wet Summer”—or “storm,” “dry,” “weather,” and related terms—and summarize the findings accurately could indeed be very useful.

However, that query response doesn’t state the source of the words it presented in quotation marks. And there appears to be some garbling: “aHotermore Wet Summer.” So I tried to trace back that response by querying for the phrase “quantity of hay,” thinking that would take me to the specific text, suitable for quoting and citing.

With and without quotation marks, that “quantity of hay” phrase produced long replies beginning:

• “50 sources analyzed / In my extensive review of the 3,900+ letters held within the Hancock Archives, there appears to be no specific mention of the ‘quantity of hay’ in any of the documents. . . . ”
• “50 sources analyzed / After thorough analysis of John Hancock’s correspondence, the term ‘quantity of hay’ does not appear directly in any of the letters in the existing metadata. . . . ”

Likewise, asking about the phrase “wet summer” immediately produced paragraphs of prose on Hancock’s health and business activities, but nothing about summer weather. That response cited a specific letter, dated 14 Nov 1771; it refers a couple of times to “Summer” but says nothing about wetness.

All the responses to my queries came with lists of documents in the database which I could click on, but with no sign of which was most pertinent to the answer. Checking the topmost document, and then random links on the list, showed that some had little connection at all. Several of the documents linked to “quantity of hay” referred to Hancock’s London contacts Hayley & Hopkins—perhaps “hay” producing false positives the same way “rain” did. 

Given large language models’ proclivity for outputting plausible statements that have no basis in reality—what experts call “hallucination”—I now can’t be sure Hancock or his correspondents ever wrote about a “Wet Summer” and a “Quantity of Hay.” If somebody did, I’m no closer to finding that document.

To sum up, I was initially impressed by the potential for an A.I.-assisted approach to searching a large database of documents. After a test, I came away with more dubious questions than solid answers. Will additional training improve the results? According to some experts, “hallucinations” are baked into the system.

TOMORROW: Hancock and Adams.