macondo

A crossword board game AI, written in Go

View on GitHub

AI Explainability

What is it?

Macondo has an experimental feature to allow you to explain a move with generative AI. If you are curious, the prompts used are in the /explainer directory.

See this position:

   A B C D E F G H I J K L M N O     ->              player1  ACDEPQU  400
   ------------------------------                    player2           440
 1|C O X A     B =   R O P I N G |
 2|  -   T   D I N G Y       -   |   Bag + unseen: (16)
 3|    - t     Z O ' A I N E E   |
 4|'     A       V     F O W T H |   E E G I I I L L N O O R R S S U
 5|      S -     A     F O E   U |
 6|  L E K E "   T   M Y     " T |
 7|    R E L A T I V E     '   I |
 8|A R E D       N       '   B A |
 9|    '       ' g '       ' R   |
10|  "       "       "     W O   |   Turn 0:
11|        -           -   H A   |
12|'     -       '       - I D ' |
13|    -       '   '       M E   |
14|  -       "       "       N   |
15|=     '       =       J U S T |
   ------------------------------

You can load it as follows:

macondo> load cgp COXA2B2ROPING/3T1DINGY5/3t2ZO1AINEE1/3A3V2FOWTH/3S3A2FOE1U/1LEKE2T1MY3T/2RELATIVE4I/ARED3N5BA/7g5R1/12WO1/12HA1/12ID1/12ME1/13N1/11JUST ACDEPQU/ 400/440 0 lex CSW24;

Running AI Explainability

To get started, you’ll need an API key from either Gemini or OpenAI:

For Gemini: Create an API key at https://ai.google.dev/gemini-api/docs

For OpenAI: Get an API key from https://platform.openai.com/api-keys

Configure Macondo with your preferred AI provider and API key:

macondo> setconfig genai-provider gemini
macondo> setconfig gemini-api-key your-api-key-here

Or for OpenAI:

macondo> setconfig genai-provider openai
macondo> setconfig openai-api-key your-api-key-here

You can also customize the AI model used:

macondo> setconfig gemini-model gemini-1.5-flash
macondo> setconfig openai-model gpt-4o-mini

Once configured, load a position and run:

macondo> explain

Additional Options

The explain command supports several options to customize the analysis:

macondo> explain -plies 3 -stop 95
macondo> explain -opprack AEINRST
macondo> explain -plies 4 -stop 95 -opprack TESTING

Options:

For more details, use help explain or help setconfig within Macondo.

Reading the opponent’s rack

explain -infer infers what the opponent is holding from the play they just made, then simulates twice — once sampling their rack from that read, once ignoring it — and explains the first while using the second to say what the read changed:

### What the opponent's last play gave away
Their last play was run through an inference: every rack they could be holding,
weighted by how likely it is that a good player would have made that play from
it. 2622 leaves came out of it, with the top three holding 9% of the weight
(effective sample size 157.2).

Most likely racks:
  EST          3.7% of the posterior
  EIS          3.4% of the posterior
  ...

Tiles they are likelier or less likely to hold than chance would give them:
  E    +0.41 tiles more than chance  (24.0% of the read vs 10.4% of the unseen pool, 7 unseen)

What the read changed:
  The recommendation is D4 BRE(A)DING either way.
  D4 BRE(A)DING wins 65.25% with the read and 66.96% without it (-1.71).

Deviations are reported in tiles of a rack, not as ratios. A tile at three times its expected share is startling until you notice that still amounts to a twentieth of a tile; the difference a player can act on is “about half a tile more vowel-heavy than random”.

None of this reaches the explanation unless it earns its place. The read has to say something — some tile has to deviate by enough to describe — and it has to have changed the analysis, either by recommending a different play or by moving the recommended play’s win% by more than a point. Statistical significance alone isn’t enough: two well-converged sims have narrow intervals, and in a position that is already won the win probabilities saturate and their intervals shrink to almost nothing, so a swing from 97.9% to 97.0% clears the statistical bar while changing nothing about how to play.

A read that changes which play is best is the most instructive case there is, and gets its own treatment: the same position has two different right answers depending on whether you believe the read.

Expect explain -infer to be slow — a 20-second inference plus two full simulations. If there’s nothing to infer from, you get an ordinary explanation and a note saying why.

Why is this better than the move I made?

When you step through a loaded game, explain compares the best play against the play you actually made on that turn, taken from the game history. Instead of “here is why this play is good” you get “here is why this play beats yours”:

macondo> load ~/games/vs_jesse.gcg
macondo> turn 12
macondo> explain
Comparing against the play you made: 15G I(L)IA (use -vs off to skip)

Your play is simulated even when it wouldn’t otherwise have made the cut, so a bad move still gets a real answer rather than a shrug, and it gets the same follow-up and board analysis the best play gets. Macondo works out the head-to-head itself - win%, equity, score, leave value, what the opponent gets back, and which follow-up chances each play has that the other doesn’t - and tells the model when the difference is not statistically established, so a coin-flip between two plays doesn’t get dressed up as a lesson.

Use -vs <play> to ask about some other play, and -vs off for a plain explanation. If the play you made was the best one, Macondo says so and compares it against the runner-up instead.

How it works

Behind the scenes, Macondo runs a full simulation and then does the analysis itself before saying a word to the model:

Because the prompt is assembled rather than written, reading the exact text that produced an explanation is the practical way to debug a bad one. explain -show-prompt prints it next to the answer, and explain -show-previous-prompt prints the last one and its response — that still works after you have moved on to another position. Both dumps are divided by ======== banners, and each banner labels everything below it up to the next one: Macondo’s own notes first, then the tool definitions, the system message and the user message exactly as sent, then the reply. The division is by section rather than by any per-line marker, because the prompt is itself markdown — a heading like ## What to say is part of the system message and really is sent.

The tool definitions are included because the request carries them in a field of their own rather than in the message text, and their descriptions are substantive: what stops the model inventing a follow-up play is a sentence in get_our_future_play_metadata’s description, not anything in the prompt. Sections run in the order the API carries them — tools, system, then messages.

One thing the dumps can’t show is the tool-call round trips. The agent SDK owns that loop, so the conversation the model finally answers from also contains its own tool-call messages and the JSON our tools returned.

Set MACONDO_NO_LLM=1 to print the assembled prompt instead of calling an API at all. explainer/example/ runs the whole thing end to end on a fixed position.

Gemini 2.5 Pro Experimental response:

This should print something like the following after a few seconds:

Model response: Okay, let’s break down this position. The simulation identifies 12K QU(ID) as the strongest play. Here’s why:

Models

Updated Sep 24, 2025 It seems you can use gemini-2.5-pro for free again (or still?).

Updated Jul 2, 2025: Gemini 2.5 Pro is no longer available for free, so we now default to Gemini 2.5 Flash, which is available for free and with a response quality that is almost as good.

At the moment of writing this (April 14, 2025) the model we are using, Gemini 2.5 Pro, is perhaps the strongest AI model out there. An experimental version of it is available for free. You should be able to run around 25 explanations per day. Normally, an explanation with this model would cost around $0.04.

You can check the Lua script above (in the scripts/lua/genai_explain.lua) to update the model. The default value is gemini-2.5-pro-exp-03-25. When the model becomes generally available, this default value is likely to change. You can also change the model used by modifying the GEMINI_MODEL API key. For example, gemini-2.0-flash is 1-2 orders of magnitude cheaper, and the quality of the response is almost as good:

Gemini 2.0 Flash response:

Model response: In this position, 12K QU(ID) performed best. This is why:

Using OpenAI

You can easily switch to OpenAI by updating your configuration:

macondo> setconfig genai-provider openai
macondo> setconfig openai-api-key your-openai-key-here
macondo> setconfig openai-model gpt-4o-mini

The OpenAI models use fewer output tokens than Gemini models since they don’t “think out loud”, and provide excellent results. The cost is typically lower per explanation compared to Gemini.

gpt-4.1 response

Model response: In this position, 12K QU(ID) is the best performing play. This is why:

In short, QU(ID) at 12K wins because it keeps your comeback options open: it gives you direct access to a strong bingo setup, keeps good tiles for more bingos, and doesn’t sacrifice average score.

Other models

You can use different models by updating your configuration:

macondo> setconfig openai-model gpt-4-turbo
macondo> setconfig gemini-model gemini-2.0-flash

Use help setconfig for more configuration options.