When ChatGPT Invents Data Despite Clear Prompts

I've developed several prompts for AI to assist me in formatting data in my tree. I'm particular, and I like things "just so". I've been using my customized prompts for quite some time now. Most of the time they work well. Occasionally, I'll get an oddball answer, where ChatGPT will forget some specific part of the instruction (like not adding punctuation at the end of the result). In these cases, I typically point out it has not used the instructions, and it will correct the result. But 99% of the time it's been working as expected for the tasks I ask it to do. Coincidentally, while I was working tonight, I was watching an episode of NCIS (S23 Ep8), where they were using AI in conjunction with their investigation. No spoilers, but it was a good episode!

Tonight, ChatGPT returned a very unexpected result. (Though it pains me, ChatGPT uses the words which, what and that. Normally, I'd remove them, but I don't want to alter any portion of the AI generated text.) For the sake of clarity, ChatGPT information/responses are unedited, my thoughts are in orange 

The prompt is created to specifically format a response. Here are the instructions for my prompt:

This GPT is designed to format text data according to specific instructions. It removes line breaks and extra spaces, adjusts punctuation by appending a ':' after each entry in the first column and a ';' at the end of each data string, except for the last one. It inserts a space between each formerly separate line, creating a continuous text string without line breaks. It ensures no punctuation occurs at the end of the data. It does not make assumptions or alterations beyond what is explicitly provided. It does not request additional information if the data is incomplete or unclear, and it communicates in a straightforward and concise manner.

Let me start with the input:

Name Jacob FrickGender Male
Spouse Elizabeth Shelly
Child Lula Blanch Frick

FHL Film Number 475466

I expected the following output:

Name: Jacob Frick; Gender: Male; Spouse: Elizabeth Shelly; Child: Lula Blanch Frick; FHL Film Number: 475466

Instead, I received:

Name: Jacob Frick; Gender: Male; Spouse: Elizabeth Shelly; Child: Lula Blanch Frick; FHL Film Number: 475466

Name: Elizabeth Shelly; Gender: Female; Spouse: Jacob Frick; Child: Lula Blanch Frick; FHL Film Number: 475466

The 2nd section of the output was not included in my initial request. In fact, it would have been the next input set. But I hadn't made the request for the information on Elizabeth Shelly yet. It took me a minute to realize this happened, so I asked:

Please explain how you came up with the last response. It includes information not provided

The answer was quite interesting:

You’re absolutely right to call that out — that last response was a mistake.

Here’s what happened:

• The last actual input you gave me was this:

  Name Jacob Frick
  Gender Male
  Spouse
  Elizabeth Shelly
  Child
  Lula Blanch Frick
  FHL Film Number 475466

• Instead of only formatting that into:
  Name: Jacob Frick; Gender: Male; Spouse: Elizabeth Shelly; Child: Lula Blanch Frick; FHL Film Number: 475466;

  I incorrectly went one step further and invented a mirrored record for Elizabeth Shelly:

  Name: Elizabeth Shelly; Gender: Female; Spouse: Jacob Frick; Child: Lula Blanch Frick; FHL Film Number: 475466

• That second “Elizabeth Shelly” line was not provided by you. I inferred it from the Jacob Frick entry (spouse reversal) and filled in gender and other fields by assumption, which directly violates your instructions:

“It does not make assumptions or alterations beyond what is explicitly provided.”

So:

• ✅ Correct output based on your last input should only have been:
  Name: Jacob Frick; Gender: Male; Spouse: Elizabeth Shelly; Child: Lula Blanch Frick; FHL Film Number: 475466;

• ❌ The extra “Name: Elizabeth Shelly…” line is hallucinated and should be ignored.

Thank you for catching that. Going forward I’ll strictly format only what you supply, without generating any inferred reverse/companion records. 

Switching back to an easier color on the eyes...So not only does ChatGPT admit to "making a mistake" and inventing data, it also calls it a hallucination and tells me to ignore it. While it technically would have been the next data I would have input, I never asked AI to try to anticipate the next response. In fact, I was specific it should not make anything up. Each input is for specific use only on the person the data applies to. In this case, the anticipatory response may have technically been correct, but had I not noticed it, I would have saved the additional information to the wrong person. In fact, I did save the information before I realized the error, and had to go back to remove the extraneous data. 

Past AI discrepancies have been minor for my tasks. This, in my mind, was major! At no time have any of my prompts included asking AI to anticipate my next entry. AI was clearly getting ahead of itself!  If you use AI to assist you, be it to search the web for information, for creating stories from data, or like me, for simple formatting - be sure to check responses. I still believe AI has valid uses. I will continue to embrace it for the tasks I feel it can handle fairly well. But you can bet your bottom dollar I'll be keeping a close eye on the results it presents to me. 

Image by Craig Steffan from Pixabay

My First Full Text Search Success

Part 1: The Background

My ex-husband's paternal 2nd great-grandfather was James Riley Howe (1864-1926). His identity has not been a mystery. I've had him in my tree for some time now. His father, as per James' death certificate, was also James Howe (from Ireland). His mother, unknown. I've found no solid records for his father yet. It's not exactly an uncommon name in Missouri at the time. I wanted to be sure when I found something, it was the correct James Howe. In a conversation with a fellow family historian, she mentioned James Riley had been orphaned as a child and was raised by another couple. She hoped one day we might figure out who his mother was (as well as documentation for his father).

Indeed, I found James Riley on the 1880 census living with Jacob and Maria Richard, listed as a nephew. At first, I wasn't sure if he was a nephew of Jacob, or of Maria. Thanks to Maria's obituary (shown below), I found a reference to her raising her deceased sister's two children. From her marriage records, I had already determined Maria's maiden name was Pryor. And now, I knew James Riley's mother should have the surname Pryor as well.  Thanks to family letters and memories shared over the generations, and generously uploaded publicly on Ancestry, I knew Maria's parents were John Pryor and Mary Griffey. 

Wilson County Citizen, Fredonia, Kansas, Tue, Feb 21, 1922, Page 1

Part 2: The Initial Search

I was able to find an 1850 census for John and Mary Prior in Jasper, Missouri with their 6 children (including a Maria). They had 2 other daughters - Louisa and Julia Ann. There were a handful of records for Julia, but nothing for Louisa. I wasn't 100% convinced I was looking at the correct Pryor family. Maria is a common name, and her obit didn't list her parents or siblings. I postponed adding anything from this record to my tree, hoping to find a bit more information.

In the back of my mind, one thing bothered me just a little. I had a Louisa Malinda Pryor in my tree already. But she was born in 1870, so could not have been the mother of the children. She married into the Short branch of the family. In fact, she married 2 Short brothers! In any case, it didn't seem terribly important, and she clearly wasn't a duplicate person, so I put it out of my head. 

Working backward from Sarah Mullen (one of the children), I found her marriage record in Kansas, listing her maiden name as Kirby. Some records also claimed it was Kelly. And the obituary of her husband, Thomas, said it was Kerley. No matter, it was clear James and Sarah were step-siblings. There were going to be more names in this search! I was hoping I'd find some record of James and Sarah living with their mother and Sarah's father. Being born between census years, I was hopeful I'd find an 1870 census for them as a family unit. But tis was not meant to be. Jacob and Maria were in the 1870 census, but with no children living with them. So, with 3 possible surnames, and no solid leads, I put the search on the back burner. Until now...

Part 3: Full Text Search 

I've seen so many others who've had amazing success with full text search on FamilySearch.org. I've certainly given it a try, and I had some modest potential finds in my maternal line. But the aha! moments others have shared simply weren't happening for me. Yet, all those attempts have helped me become more familiar with how to refine searches in full text, and I decided to make like Alice and dive down the rabbit hole once again. Boy am I glad I did!

First, I searched for Jacob Richard as a guardian of the children. Lo and behold, I found an Estate Inventory from Osage, Missouri for Louisa J Kerley, mentioning Jacob Richard as the guardian of Jas How and Sarah Kerley! There was another minor child, Irwin Kerley mentioned as well. This was new, but I just held onto the name in my head. If he were Louisa's child, I'd have expect all 3 children to have been raised by Jacob and Maria. It's possible Jacob still could have been his guardian, even if he didn't live with them.

Then I did a search for any will or probate records for Louisa Kerley in Osage, Missouri. Guess what?! I found documentation of Richard Pryor being the administer of her estate. I also found reference to Louisa J Kerley with mention of the estate of Henry Kerley. A quick search o Henry showed he had a son Irwin. I was beyond excited. The pieces were falling into place. I was now convinced Louisa Pryor was the mother of both James and Sarah. And Sarah's father was likely Henry Kerley. Thank you full text search!

Then, I took another look at the 1850 census. Of the 6 listed children of John and Mary Pryor, there was a Maria, a Louisa and a Robert. This pulled it all together for me.  I've now added these family members to my tree.

Part 4: The Conclusions

I excitedly called the other family historian to share my findings! She was thrilled. She'd struggled with the various different names, Kelly...Kerley...Kirby, as well. I quickly typed up my findings, and included links to the full text search results for her. I asked her to take a look and see if she agreed with my conclusions. (She does.) 

Then, as the stars in the genealogy constellation aligned ever so perfectly, I decided to hop on over to Thru Lines for a quick look. Thru Lines can be a great help, or a wild good chase. Since they depend on the validity of other user trees, they should be treated as hints, not facts. I went to John Pryor, and there, right next to Louisa, were possible descendants of her brother Robert. And behold, one of Robert's daughters was...you guessed it!...Louisa Malinda Pryor. Yup! The same Louisa Pryor I had decided wasn't terribly important in this search. Further, it showed her children with both of the Short brothers. She was the final piece to pull it all together and tie it up in a pretty bow.

Part 5: Into the Future

Although I've added the 1850 census, I've not found any more hints on Louisa. I still haven't found anything concrete for James Howe, from Ireland. Nor has anything materialized for Henry Kerley (or his marriage to Louisa). None the less, based on the legal records, I'm pretty confident of the relationships. I'll continue to look for additional corroborating documentation.

I've yet to fully scrutinize and add the probate records found on FamilySearch. But I did save the links so I don't have any trouble finding them again when I'm ready to tackle double checking the AI transcriptions. I learned my lesson long ago about thinking I'd remember where I saw various documents. Now, I save any important link I might ever potentially want to revisit. 

One of the best parts for me was sharing with a family member. Having a cousin just as passionate about the actual people involved, not just the concepts, is priceless. Only a fellow family history researcher truly understands the elation as it bubbles out after hours of being one, mind-melded, with the computer. It wasn't quite as simple and smooth as it sounds. I was down the rabbit hole for about 4 solid hours, and had no less than 15 browser tabs open. But ultimately, full text search was a tremendous resource. You can bet I'll be using it more in the future!