TL;DR​

• N.D./S.D. can mean “Northern District” and “Southern District” for federal cases, overlapping with “North Dakota” and “South Dakota” postal abbreviations; other states have possible issues here like MD and OK
  • “CA” (Court of Appeals? California?). I addressed this in my “Writing Help” game.
  • “SC” is a triple threat: South Carolina, Supreme Court, or Superior Court?
• I looked at the data. Yes, there were some human errors, but it is clear that Charlotin has put in the work to create and maintain this valuable resource. That’s why I think people find it worthwhile to send him cases. He’s provided something of value and put in the time. Anyone who looks closely at the data can see that. People who churn out slop reports, on the other hand, are wasting your time. If it wasn’t worth their time to write, why would it be worth your time to read?
• Once you account for relative population, the map is much more of a dog bites man story. To me, the more interesting part is the specifics of each case and the citation graph of cases that are frequently cited by later cases in these disciplinary decisions.
  • For example, I thought this may have been a mistakenly coded jurisdiction but it was not: Lowery Wilkinson Lowery, LLC, et al. v. State of Illinois, et al. (E.D. Oklahoma 2025). Instead, it’s an interesting attempt by the plaintiff to apply the “Indian Country” McGirt ruling and the plaintiff also misattributed that U.S. Supreme Court ruling to the Eastern District of Oklahoma Judge.
  • “There is no one righteous, not even one.” After data cleanup, New Hampshire alone seemingly had no hallucination cases, but I found both that New Hampshire had had its own hallucination case around July 2025 per reporting last fall, and that a New Hampshire attorney had been responsible for hallucinations (actually from the client) in a case in the Vermont Supreme Court.

“N.D.” and “S.D.”: Did Not (Usually) Mean the Dakotas​

Damien Charlotin maintains a well-known public database of AI hallucination cases around the world. It has even been cited in a legal case here in Iowa. I’ve written about his work before, including his point about low false-positive rates potentially creating automation bias, lulling users of legal AI tools into a false sense of security. Recently I saw a post on LinkedIn, in which Charlotin, reacting to a report by Straight Arrow News on a study by “Laine AI,” criticized the claim that AI-driven legal hallucinations are especially concentrated in North Dakota and South Dakota. Charlotin pointed out that this was most likely due to a misinterpretation of “N.D.” and “S.D.”—common abbreviations in federal cases indicating “Northern District” and “Southern District.”

The June 19, 2026 article noted how surprising the result was, with the headline: “Where does the most AI legal slop come from? Not from the states with the most lawyers.” Laine AI, according to the article, stated that North Dakota courts had 109 cases with AI errors, South Dakota had 82, and California had 59. Supposedly, this was from Charlotin’s website, but as I often advise clients, this is not a good use of LLMs: pivot tables already exist!

After I manually reviewed and coded the states for 1,145 USA cases and cleaned up data from Charlotin’s website downloaded on June 21, 2026, I found that the actual count was: California 121, North Dakota 6, and South Dakota 1. California had some “E.D. Cal” and similar, as well as two Ninth Circuit cases that were appeals from California cases, a case that was “CA SC” (Court of Appeals, South Carolina? California Superior or Supreme Court?) which was actually California; and one “CA 8th Circuit (Bankruptcy)” a federal Eighth Circuit case that was an appeal from District of North Dakota Bankruptcy.

None of these minor ambiguities would explain the large discrepancy in counts. My review of the numbers suggests that Charlotin was correct about the likely cause of the hallucinations.

You can read Charlotin’s own account of the mix-up on LinkedIn.

That being said, 6 cases in North Dakota relative to its population size and number of attorneys does seem to be somewhat disproportionate. There, I’d argue that almost every state is probably undercounting/not catching AI hallucinations, or the hallucinations have been caught, but not reported and added to the database. In fact, this exercise led me to find some information about New Hampshire as the only (apparent) state without hallucination cases.

So, Midwest’s honor defended. Let’s move on to some other lessons.

It’s not just “garbage in, garbage out”​

Sometimes you’ll hear about “garbage in, garbage out” (GIGO). As it is applied to practical deployment of generative AI in businesses and other organizations, there’s a common refrain that “the AI is only as good as the data you give it.”

I’d somewhat reframe that to say “the data quality limits how well your AI can perform.” Even with good data—company policies, solid accounting spreadsheets, detailed written reports—the LLMs can still hallucinate. LLMs can make ungrounded statements even when they have access to the correct answers, they can misunderstand details, and they can improperly combine text, among other failure modes.

Charlotin’s dataset is carefully maintained as far as I have seen, and I’ve looked through a lot of it. His database appeared to be the Laine study’s sole source of data. The Laine summary stats quoted in the aforementioned report, on the other hand, suggest AI hallucinations. These uses of AI: “AI can write this research paper,” “AI can write that quarterly report for the board,” “AI dashboards give the CEO unmediated access to perfect visibility into the organization” are actually organizationally destructive uses of LLMs. They are truthiness machines, but once you look under the hood at the data, you can tell that the summaries bear no relationship to the underlying data. Since there is no relationship, correcting the data is pointless. That’s the difference between a human-curated database—like Charlotin’s—and a slopped together dashboard or report or slide deck. Of course humans make mistakes too, and human-curated databases will have rough edges or errors, but it is worthwhile to look at them closely and help fix them because somebody who cares about the truth is on the other end.

Abbreviations Are Not Always OK​

North Dakota and South Dakota are not the only examples. There is a whole class of collisions between legal shorthand and two-letter state postal codes. Here are the ones I’m aware of:

Literal String	What it really means	Collides	Impact
‘N.D.’	Northern District	North Dakota	Low population, many federal northern districts, dramatic false spike
‘S.D.’	Southern District	South Dakota	Low population, many federal southern districts, dramatic false spike
‘M.D.’	Middle District (e.g., M.D. Fla.)	Maryland	More populous, fewer middle districts, mixed effect
‘Ct.’	Court (Sup. Ct., Ct. App., Dist. Ct.)	Connecticut	possible, but less frequently seen “in the wild”
‘CA’	Court of Appeals	California	Large population, hides in the noise
‘S.C.’	Supreme Court or Superior Court	South Carolina	Mid-spectrum

Supreme Court or Supremest Court?​

‘S.C.’ for Supreme Court raises a deeper problem. You still don’t know which court is highest in the given state without domain familiarity, because state court naming is not uniform across jurisdictions and “Supreme Court” is not the highest court everywhere:

• In New York, the “Supreme Court” is the trial court and Court of Appeals is the highest.
• Texas and Oklahoma each have two highest courts: Supreme Court (civil) and Court of Criminal Appeals (criminal).
• Maryland changed the naming to be more familiar in December 2022.

If there are other examples, please let me know. But my main point is that you can’t just assume based on the name.

Divided Islands​

When parsing strings for the name of a state, “Hawaii” or “Hawai’i” can be written with or without the apostrophe, which can also be written as ʻokina (Unicode U+02BB), a real letter in the Hawaiian alphabet. This is actually an example of an area where appropriate use of LLMs can help, because it can be exasperating for regular expressions, especially with an unpaired apostrophe.

Look at the Data and Learn About the Domain​

Anyone who has spent time with both LLMs and legal citations should recognize that “N.D.” could have multiple, conflicting meanings. Also, you should assume every map is secretly a map of population until proven otherwise. So when “North Dakota” has far more AI misuse than California, look at the rows in your spreadsheet first.

My own background is in financial-crimes investigation, OSINT, and GIS. I have hit similar failures with AI, law, and geography before, as I wrote about in December 2025 (NOTE: correction, I did this in December, but I wrote about it in January). I rebuilt my hallucination-case map with Claude Code, and at first it produced a patchwork quilt because it matched county names across states without using the state as part of the key, and there are a lot of Polk Counties. The output was obviously wrong to me because I looked at the output, and I also knew why it failed, because I’m familiar with the domain. Claude Code also tried to fabricate an Eastern and Western District of South Carolina, which is a more specific hallucination that would require domain knowledge (which I had developed), to recognize that the output was incorrect.

Nice Try, New Hampshire​

After manually reviewing and cleaning/hand-coding the state data, the only state that (apparently) had no attorneys with hallucinated cases was New Hampshire.

However, I found that a New Hampshire attorney included a fake quotation from a real case in a Vermont Supreme Court divorce case provided by his client. This touches on multiple topics I want to write about, including: a) that LLMs can hallucinate quotations, so checking that a case merely exists is not sufficient; b) that client-provided AI-generated fake citations are a problem for attorneys who “don’t use AI,” so they need a reality check about Shadow AI use; and c) family law is coming up frequently for AI hallucinations (e.g., early Iowa misuse and a recent Nebraska Supreme Court case).

It wasn’t adding up. A New Hampshire attorney representing a divorce client filed a legal brief that quoted from a previous court case. But when Vermont Supreme Court justices went to the case he referenced, the quote was nowhere to be found. In a November hearing, they asked the attorney where the quote came from. “Your honor, my client used an AI, um, helper” said the attorney. Justice William Cohen, now retired, responded: “The secondary source was AI? And you didn’t identify it?” “I’m not familiar with what’s involved with it and so forth,” the attorney said. He claimed that his client offered to write the brief using artificial intelligence. The quote came from “AI GPT or something like that,” he said later on. “I didn’t use it exactly but it’s a common one, I believe.” After the hearing, justices on the state’s highest court chastised the attorney for his mistakes in a court filing, requiring him to file a copy of the write-up in all of his pending cases in Vermont Superior Court. Valley News

Later, I found a more direct example of a New Hampshire case. The “Windham case” with Judge Lisa English, reported in October 2025 involving a NH/MA attorney. I unfortunately could not track down the name of the case, despite having the attorney’s name and the judge.

The Laine report produced unsupported conclusions about supposedly excessive AI misuses in North Dakota and South Dakota. However, my analysis of the data allowed me to find the one missing gap and show that there is no state without any AI hallucination cases.

Charlotin’s Map on July 2, 2026​

As of today, there is no state with 0 hallucination cases on the map of USA cases on Charlotin’s website.

Indian Country Jurisdiction: Plaintiff also claims that Judge White wrote the McGirt opinion (he did not)​

Because I was manually reviewing the cases for state coding, I thought that a case with “State of Illinois” as Defendant, but in the Eastern District of Oklahoma, may have been mistakenly coded. It was not. Instead, it was interesting case and a great example of why you should look at the data and not just let AI summarize everything in the aggregate for you.

Lowery Wilkinson Lowery, LLC, et al. v. State of Illinois, et al. (E.D. Oklahoma 2025).

Multiple times, the plaintiffs attempted to raise several jurisdictional arguments under McGirt that are so inadequate and clearly unresearched that any reasonable attorney would know they lack merit. Plaintiff repeatedly asserts that if this court fails to direct bar disciplinary proceedings in Illinois, that it is effectively “overturning” McGirt v. Oklahoma, 140 S.Ct. 2452 (2020), a case where the United States Supreme Court held that the entirety of the Eastern District of Oklahoma is “Indian Country” for the purposes of the Major Crimes Act. Dkt. No. 74 at ¶44. Plaintiff also claims that Judge White wrote the McGirt opinion (he did not), indicating that counsel has not made any attempt to research the case. Id. In addition, plaintiffs further allege that “the Defendants accused Plaintiff Lowery of a violation of the Major Crimes Act while on an Indian reservation land under McGirt … Therefore, if the Court wants to exercise jurisdiction under McGirt it would be Mandatory jurisdiction.” Lowery III, Dkt. No. 22 at p. 5. OMNIBUS ORDER

Takeaways​

• Even if an LLM feels like it gives you an overview of the data, look at the data. There’s fun stuff in there and you’ll catch more hallucinations.
• But you should also ask if you should even be using LLMs for a given data analysis task. Pivot tables and SUM functions still exist.
• Your starting assumption should be that every map is a population map.
• Damien’s database is legit.
• Come at the Midwest, better not miss.

This first week of January, the general feeling is very much everyone bringing out the winter vacation vibe coding projects cooked up on Claude Code. Claude Code itself isn't new, but with Opus 4.5 being so much more powerful, something just clicked for a lot of people (myself included). For me, it turned a lot of "when I have a couple days" projects into "well that's done, let's do another."

I am mainly going to describe in this post how I updated the map for my website, along with the hallucinations I saw along the way. I'll also talk about how prior programming experience and domain expertise in geographic information systems (GIS) helped with dealing with these hallucinations.

But first, I wanted to tick off a few other projects I did recently, just since my end of 2025 post.

• I updated my transcription tool to support many more file types than just MP3 and added a GUI.
• I got Claude Code to completely modernize Taprats, a geometric art Java program from Craig S. Kaplan. It appears to work just like the original so far, but I'll test it more before writing about it.
• I built a local LLM spoiler-free summarizer of classic books. It increments to the chapter you left off on.

And more stuff. It's very exciting. I get why people are work up about Claude Code.

But that's why it's important to be reminded of hallucinations. Not to dunk of Claude Code, but to keep people grounded and maintain skepticism of AI outputs. You still have to check.

Safety First​

I do not dangerously skip permissions. I know it can be exciting to get more out of AI agents. But the more agency you give it, the more harm it can do when it either goes off the rails or gets prompt injected to be a double-agent threat.

Claude's Hallucinations​

• Opus 4.5 hallucinated that there were two federal districts in South Carolina to fix an undercount.
• Mixing up same-name counties (not exactly a hallucination, actually a common human error).
• Claude removed Yellowstone National Park, a few military bases and a prison from the map (rather than shifting district borders from one district to another).
• "Iowa Supreme Court Attorney Disciplinary Board" shortened to "Iowa Supreme Court," making it sound like an Iowa Supreme Court case.
• I previously tried to used the tigris GIS package in R as source of a base layer of U.S. District Courts, but Opus 4.5 hallucinated a court_districts() function (this was not in Claude Code).

The South Carolina Counting Hallucination​

I used Claude Code to build the Districts layer from counties and states based on their statutory defintion.

Claude Code with Opus 4.5 didn't initially hallucinate about the District of South Carolina. Rather, when I went back to make some edits and asked Claude Code in a new session to check the the work in that layer, it counted and said there should be 94 districts, but there were only 91. The actual cause of the error was that the Marshall Islands, Virgin Islands, and Guam were excluded from the map.

Claude said "let me fix that" and started making changes. Rather than identify the real source of the undercount, Claude interpreted that as just an undercount. So Claude tried to make up for the undercount by just splitting up districts into new ones that didn't exist.

Claude split South Carolina in two and started to make a fictitious "Eastern District" and "Western District" which do not exist. But if you just wanted a map that looked nice without actually having familiarity with the data, then you might go along with that hallucination. It could be very persuasive. But actually the original version with just District of South Carolina was correct. South Carolina just has one district.

Patchwork Counties​

When I had initially created this districtmap, it looked like a quilt. It was a patchwork of different counties wrongly assigned to different districts.

I don't know specifically why different areas were assigned to the wrong districts. I think primarily the reason is because there are a lot of same-named counties that belong to different states. So, probably Claude was just matching state names and then kept reassigning those states to different districts.

For example, Des Moines is in Polk County in Iowa. But there are a lot of Polk counties around the country. So if you're not using the state and county together as the key to match but you're just matching along the single dimension of using the county name, then you would have a lot of collisions. That's something that I'm very familiar with working with GIS.

If somebody were not familiar with GIS, they wouldn't really necessarily suspect the reason why, but it would be obvious that the map was wrong.

Since I was able to pretty quickly guess that that might have been the reason, I suggested a fix to Claude. That fixed most of the issues with most of the states.

Uncommon Problems with the Commonwealth of Virginia​

One of the issues that was still persistent when I was building the districts from county level was in Virginia. I've actually lived in Virginia, so I was familiar with the city-county distinction. They have independent cities that are separate from the counties if they're sufficiently large and have a legal distinction from the surrounding county. For example, Fairfax City and Fairfax County are distinct things. It's even more confusing, because the school districts go with the counties. Most states don't follow that.

So I had to get Claude Code to wrangle with that. Claude even reviewed the statutory language. I could tell from reading as Claude was "planning" that it considered the Virginia city-county challenge, but it still failed on the initial attempt.

I had to iterate on it multiple times. I had to tell it that it had missed out on a whole area around Virginia Beach. It had flipped a couple cities and counties where it appeared that there was a city that had a similar name to an unrelated county in the other district. Claude just assumed that all counties and cities that had the same name were in the same location and assigned them the same. Then it had to go and look at where they actually were located and then reassign them to the appropriate Eastern or Western District.

But eventually I got to a point where it had good districts for Virginia.

Wyoming (and Idaho and Montana) and North Carolina​

Now there are a couple other weird wrinkles in Wyoming and North Carolina. They don't follow the county boundaries completely.

Wyoming is the only district that includes more than one state. District of Wyoming also includes all of the parts of Idaho and Montana that are in Yellowstone National Park.

For North Carolina, rather than completely following county boundaries, there are a couple of military bases and a prison that are across multiple counties where the boundary follows the lines there rather than the county lines.

Initially I ignored those wrinkles. But once the rest of the map was in good shape, I just wanted to see what Claude could do.

I explained those issues and asked Claude Code to see if it could clean those lines up and get a map that reflected those oddities.

It did on the second attempt. But on the first attempt, Claude ended up just cutting out Yellowstone National Park and those military bases and that prison from any district. So there were just blank spots where Yellowstone would be that was just cut out of Idaho, Montana, and Wyoming. Those bases and that prison were just cut out of either the Eastern Districts or Middle District of North Carolina.

That was a problem, obviously, because they needed to be shifted from one district to another, not removed from all districts. So I needed to explain more specifically what I wanted Claude to do to fix that. It needed to move the lines, not to remove them entirely from the map. That second attempt got it cleaned up.

Claude Still Saved A Lot of Time Accounting for Hallucinations​

And I was still very impressed with Claude doing that. But having familiarity with the data and looking at the output were important.

There's no doubt in my mind after doing all this that Claude saved a tremendous amount of time compared to what I would have had to do with manual GIS workflows to get this kind of a map on a desktop computer.

Then there's another layer of having it be responsive in all the ways that I needed it to be on my website for other users. So it is just tremendous to see how cool that is.

But I do think that domain expertise, familiarity with GIS in the past was still helpful to me, even though I didn't have to do a lot of hands-on work. Just being able to guide Claude through the mistakes that it made and being able to check the output was very helpful. Since it's a map, since the output is visual, there were some things that anyone could see, obviously, that it got wrong. Even if you didn't know why it might have gone wrong, you could tell that the map was wrong. And you might have been able to get to a better finished product by iterating with Claude Code. But you might have also wasted more time than I did with Claude if you hadn't had GIS experience to guide your prompting.

Map Features with Claude Code

Use Github, Try to Keep Formatting Code Separate from Text/Data​

I had already written this, and I stand by it.

However, as powerful as Claude Code is, it is also important to use GitHub or something similar for version control. It is also critical to make sure Claude is changing code but not your actual writing.

Claude Code and My Map with Links to Blog Posts About AI Hallucinations Cases

This map is not a map of every AI hallucinations case, but rather every case that I have blogged about so far. Basically, it's federal and state cases where there has been either a strong implication or the direct assertion that there was AI misuse. Many of these cases cite Mata v. Avianca.

Lone Case Markers​

If you click on a given case and it's a single case, you'll see:

• what the case is called
• the year
• the jurisdiction
• the type of case (federal or state), which is also indicated by the color
• links to related articles where I've talked about that case

Clusters, Spiders, and Zooming​

Getting the "spiderize" functions to work was the must frustrating part of all of this. I made several prior attempts with Claude Code on Opus 4.5. With the same prompts, this most recent attempt finally just worked on the "first" attempt (of that session). I only tried again an afterthought once all the other features were done. But previously, I'd wasted a lot of time trying to get it right. So both a Claude Code success and faillure. Still, I'm happy with the final result.

Zoom to Mata v. Avianca​

If you click those links, it'll jump over either to my company blog or the Substack articles where I've talked about those cases.

Additionally, if they reference other cases that are also on the map, such as Mata v. Avianca, then there will be lines drawn from the case you clicked to the other cases on the map reference or are referenced by those other cases. The map will give you a little count summary at the bottom: "Cites three cases" or "cited by" so many cases.

So if we look at Mata v. Avianca, the marker is not by itself on the map. If you look at the eastern United States from the starting zoom level that I'm looking at as I'm writing this, you see a "4." The 4 has a slash of red and orange, meaning there are both federal and state cases.

If you click the 4, the map zooms in. Now there are three over the New York-New Jersey area, and one over Annapolis, Maryland.

Click the three, and the map zooms in further. That splits between one in New Jersey and two in New York.

Click the two, and then those two "spider out" because they are both in the same jurisdiction. One is Mata v. Avianca, and that is cited by fourteen cases currently. It's a 2023, Southern District of New York, federal district court case. The other is Park v. Kim, a 2024 case, which is actually a Second Circuit Case that is placed on the map in the same location.

The New Jersey case is In re Cormedics, Inc. Securities Litigation, a 2025 case from the District of New Jersey, which is a federal case, and that was one of the cases that was discussed by Senator Grassley asking judges about their AI misuse.

Other Clusters in Mountain West, Texas​

Spider over Iowa​

So if you zoom out, you know, it combines nearby cases. If you zoom out far enough, it will combine Wyoming and Colorado, for example, or multiple districts in Texas. But as you zoom in or as you click, it will zoom in further and split those out.

If you look at Iowa, there are five currently, and those will all spider out because they are all in the same location. But then you can click one of the individual ones and get the details.

District Level​

If you hover your mouse of a district, it will tell you how many federal cases were in that district and have a blog post about them.

Circuit Level​

If toggle off the district boundaries and toggle on the circuit boundaries, and federal cases are still toggled on, hovering your mouse over the circuit will give you a count of how many cases were in that circuit and have a blog post about them.