How the archive was collected, checked and analysed

Two Jupyter notebooks read public listing pages on The American Presidency Project and then each linked page, with browser-like headers, connection pooling, retries with exponential backoff on 429 and 5xx responses, and fixed delays between requests.

Three functions handle dates, and they disagree on purpose. The listing scrapers turn the ISO content attribute into “September 29, 2024” with strftime("%B %d, %Y"), keeping the wall-clock date of the timestamp rather than converting time zones. The content extractor goes the other way: an ISO string containing both “T” and “+” is kept as ISO, and anything else is tried against four formats in order (%B %d, %Y, %m/%d/%Y, %Y-%m-%d, %d %B %Y) before being passed through unchanged. A trailing “Z” therefore survives in the content phase but not in the listing phase.

For each paragraph whose first bold element starts with “participants” or “moderators”, the notebook takes the paragraph's HTML, removes a leading <b>LABEL:</b>, replaces each <br> with a sentinel, strips tags, splits on the sentinel and then on semicolons, and trims trailing full stops. That is why names keep a trailing “and”. The plain text is every text node, stripped and joined with newlines, with runs of three or more newlines collapsed. The TypeScript port does the same over an htmlparser2 tree, including BeautifulSoup's serialisation (<br/>, minimal escaping) and Python's definition of whitespace. One known gap: malformed HTML (an unclosed or nested <p>) is repaired differently by htmlparser2 than by Python's html.parser, so a pasted fragment like that can split differently in the playground. All 179 stored transcripts are well formed and match exactly.

Documents are filed under one candidate, but interviews and town halls also contain interviewers, moderators and audience members. Before counting words, each stored document is split into its paragraphs and read in order. A paragraph that opens with an upper-case label (“ROGAN:”, “AUDIENCE MEMBER:”) switches the current speaker until the next label. Labels that contain the candidate's surname or an office they held (“THE VICE PRESIDENT:”) keep the text; labels that are fields of a press release (“FACT:”, “TIME:”, “NARRATOR:”) are not speaker changes; any other label drops the text that follows. Stage directions such as “(Applause.)” are removed.

This keeps 94.5% of all words and removes 265,530 words spoken by other people. The original word counts are untouched and are what the Explorer reports; the cleaned text feeds the reading grades, the word index and the timelines.

The Flesch-Kincaid grade level combines average sentence length and average syllables per word. Words are runs of letters (with internal apostrophes); sentences end at “.”, “!” or “?” followed by a space or quote, after protecting titles (“Mr.”) and initials (“J.D.”); syllables use a standard vowel-group heuristic. Texts under 100 words get no grade. Transcribed speech and press releases are punctuated by different people, so grades compare like with like only roughly, and they say nothing about the quality of what was said.

The comparison uses the weighted log-odds ratio with an informative Dirichlet prior from Monroe, Colaresi and Quinn (2008), “Fightin' Words: Lexical Feature Selection and Evaluation for Identifying the Content of Political Conflict”, Political Analysis 16(4). The prior is the whole archive's word distribution scaled to α₀ pseudo-words (10,000 by default): it pulls rare words towards the archive-wide rate so that a word used twice by one group and never by the other does not top the list.

The index holds 16,877 words: lower-cased tokens of two or more letters, present in at least five documents, minus a standard English stop-word list. Counts per document are stored as compressed postings; group totals are summed on the server for each request.

A period's rate is mentions ÷ words × 10,000, where words are the cleaned own-voice tokens of every document in the period. Treating the count k as Poisson, the exact (Garwood) 95% interval comes from gamma quantiles, computed in TypeScript with a regularised incomplete gamma function and checked against SciPy.

The topic list has 44 entries chosen to cover policy areas that campaigns of both major parties talk about, and both party names. Topic words are matched as whole words, ignoring case, with one exception. The two party topics are defined the same way: the party's noun and its party adjective, counted only when capitalised (“Democrat”, “Democrats”, “Democratic”; “Republican”, “Republicans”). Capitalisation separates “Democratic Party” and “Republican nominee” from the generic words “democratic” and “republican”, which are left out for both. Topics: Economy, Jobs, Inflation, Taxes, Wages, Middle class, Debt and deficit, Trade and tariffs, Manufacturing, Energy, Climate, Health care, Medicare, Social Security, Education, Housing, Infrastructure, Immigration, Border, Crime, Police, Guns, Abortion, Opioids and fentanyl, COVID-19 and pandemic, Veterans, Military, Terrorism, China, Russia, Ukraine, Israel, Iran, Democracy, Freedom and liberty, Constitution, Supreme Court, Corruption, Families, Women, Children, Faith, Democrats (party), Republicans (party).

Transcripts from six decades mark speakers in different ways, so a new turn starts at a bold or italic label (“WALZ:”, “The President.”), a plain upper-case label ending in a colon (“MR. NIXON:”), or, in transcripts that use neither, a short name ending in a full stop (“Mr. Newman.”). Lines without a label continue the current speaker. Labels are reduced to a surname; “THE PRESIDENT” and “THE VICE PRESIDENT” become the office holder of that year.

A speaker is a candidate if the surname is on the list below of people who took part as candidates in that cycle's debates (public record), belongs to the party holding the debate when it is a primary, and is named in the page's Participants block when there is one. Every other named speaker is grouped as a moderator, panellist or questioner; audience members, unidentified voices and recorded clips form a third group. Word share stands in for talk time.

Recorded material played during a debate is not live speech. Turns between “[begin video clip]” and “[end video clip]” (and the transcripts' other spellings of these markers), turns tagged “(from videotape.)”, and labels such as “VIDEO CLIP OF …” are counted as Recorded clips, so a candidate heard only in a clip, such as a president quoted at the other party's primary, is not listed as taking part. A clip whose end marker is missing covers only the next speaker. A bold “MODERATOR:” label is a turn unless it heads the list of names at the top of the page, and the build stops if moderators get less than 2% of a debate's words, which would mean their labels were missed. Where a source transcript leaves out a label, the words go to the previous speaker; one such gap in the January 2004 Greenville debate is visible in its moderator share.

Source. Gerhard Peters and John T. Woolley, The American Presidency Project, University of California, Santa Barbara. Copyright © The American Presidency Project. Document and debate pages are linked, never re-published. The site stores titles, dates, counts and short quotations of at most 25 words, each linked to its page.

Coverage. The documents are what the archive files under “campaign documents” between 2016-01-01 and 2024-11-06; that is not everything a campaign published, and the volume per candidate reflects both the campaign and the archive. 26 listing rows were duplicates and are counted once; 46 documents have no text.

Neutrality. Every speaker is processed by the same code. Colours identify the groups being compared, never parties. No measure here is a sentiment, quality or truthfulness score, and nothing predicts an outcome.

Code. MIT licence, covering the code only. The original notebooks and CSVs are kept unchanged in original/ of the GitHub repository.

The Fightin' Words z-score treats each word token as an independent draw. Campaign text is clustered: one press release can repeat a county's name thirty times. Every comparison on Distinctive words is therefore checked by resampling documents, not words: the documents of each group are drawn with replacement (200 times, seed 20261010, group A first, then group B, from one mulberry32 stream), every z-score is recomputed with the same prior and α₀, and for each listed word the page reports the share of resamples in which it stays in its side's top 30, the 2.5th to 97.5th percentile of its z-score, the documents that use it and the share of its uses from its heaviest document.

With 200 resamples a kept share has a Monte Carlo standard error of at most 3.5 percentage points. Separately, a comparison tests every word used by either group (up to 16,877), and at |z| = 1.96 about 5% of the words tested would cross the line even if the groups did not differ; the page shows that figure for each comparison. The informative prior makes it a rough guide, but the point stands: the lists are rankings to explore, not a set of findings. Rationale and results: DR-003.

The rows behind Readability are not independent. A campaign's documents share writers and a house style, and the debates of one election cycle share candidates and a transcription source. Resampling single documents or debates would treat them as independent and give intervals that are too narrow, so the intervals resample clusters (a cluster bootstrap, 10,000 resamples, seed 20261010): for the mean grade of a cycle and kind of text, whole speakers with all their documents; for a debate trend (the slope of a least-squares line per decade, and a pointwise band for the fitted line), whole cycles with all their debates. Cells with fewer than 5 documents or 10 speakers get a point estimate and no interval: with fewer speakers the cluster bootstrap runs narrow (DR-008).

The difference is not small. For 2024 written releases (1,395 documents from 10 speakers) the speaker-level interval is 10.00 to 12.92, several times wider than resampling documents would suggest. For the primary-debate trend (130 debates in 9 cycles) the cycle-level interval is −0.89 to −0.17 grade levels per decade, against −0.65 to −0.26 if debates were independent. With only 9 to 14 cycles, a cluster bootstrap is itself approximate and can still run a little narrow. A cycle's own mean in the table resamples that cycle's debates and describes that cycle only.

Because the grade is linear in words per sentence (WPS) and syllables per word (SPW), a difference of mean grades splits exactly into a sentence-length part and a word-length part. For the 14 speakers with enough of both kinds of text, each speaker is one paired difference, and the mean gap is −5.5 grade levels (95% t interval −6.9 to −4.1; at n = 14 a percentile bootstrap gives the slightly narrower −6.7 to −4.2), of which −3.2 comes from sentence length. 14 of 14 speakers grade lower when transcribed (exact sign test p = 0.0001). These intervals cover sampling, not measurement: they do not include the effect of who transcribed a debate, which the page shows separately with two events the archive holds in two transcripts. The choice of clustered intervals is recorded in DR-006.

Question. On one-sentence campaign excerpts, how often does a language model give the same policy-topic label as a careful coder, compared with a transparent keyword dictionary?

Set. 120 sentences of 12 to 25 words, 40 per cycle, one per randomly drawn document (seed 20261010), after the quotation rules. The draw is blind to keywords, so it does not favour the dictionary. Gold labels: AI draft, not labelled by a person, so every score is provisional. The gold labels were drafted by the AI coding assistant (Claude) that built the 2026 upgrade, as a single annotator, and no person has labelled them. Treat every score as provisional. Because the draft came from the same model family as the default labeller, agreement with Claude models may be flattered. The same assistant also wrote the keyword dictionary shortly before, so the gold labels are not independent of the baseline either, which could move its scores in either direction. The labels will be replaced by a blind relabel by people who have not seen the draft, not by a review of this draft.

Labellers. The keyword rules, keywords-v1 (10 October 2026), were written before the set was drawn and then frozen. The model sees the codebook (21 CAP-style topics and “none”), five coding rules and up to ten excerpts with opaque ids per request; the system prompt is 4,284 characters. Defaults: Claude Haiku 4.5 at temperature 0, or Claude Sonnet 5.5 at low effort, or an OpenAI model (gpt-5-mini by default).

Metrics. Agreement with gold (Wilson interval), Cohen's kappa (percentile bootstrap over excerpts), agreement on the excerpts with a policy topic, and a paired comparison on the same excerpts: the difference in agreement and in kappa (paired bootstrap, so both labellers see the same resamples) and McNemar's exact test. The keyword baseline on all 120 excerpts: 75.8% agreement (95% CI 67.4 to 82.6), kappa 0.59 (0.45 to 0.71).

What AI does here. One optional feature: on Topic labels, a language model you choose labels the policy topic of short excerpts so that it can be compared with keyword rules and gold labels. It runs only when you start it, with your own API key.

What it never does. It produces no number anywhere else on the site; every other page is computed without AI. It is never used to compare candidates or parties, and its labels are never presented as facts: every output carries an “AI-generated” label, and the simulated demo is labelled as simulated. A request that fails (a rejected key, a network or rate-limit error, or pressing Stop) is left out of the scores, never counted as the model's wrong answer.

Data sent to the provider. The codebook, the coding rules and the excerpts (one sentence of 25 words or fewer each, with an opaque id), from your browser straight to Anthropic or OpenAI. The speaker, date, title and link are not sent as metadata, but 35 of the 120 excerpts name the candidate in the text, and some name journalists or officials, so the model can often tell whose campaign wrote them. Your key is kept in your browser (sessionStorage, or localStorage if you tick “remember on this device”), never sent to this site's server and never logged. The provider's own terms apply to what you send it.

Human in the loop and audit. You can accept, correct or reject each run, and accept or reject any single call later from the log; corrections are recorded as edits and never change the scores, which always use the model's own labels. Each decision is added to the call's decision history, so a change of mind never erases the earlier decision, and each record holds the generation settings the request was sent with. Every call, failure and simulated run is logged in your browser's IndexedDB with the prompt, the answer, latency, token use and your decision, viewable and exportable (JSON or CSV) on the AI audit log.

Frameworks. This design is informed by the Australian Government's policy for the responsible use of AI in government, the transparency principles of the EU AI Act and the NIST AI Risk Management Framework. It does not claim compliance with any of them.

AI in building the site. The 2026 upgrade was built with an AI coding assistant, which also wrote the keyword dictionary and prepared the draft gold labels (see DR-007, DR-004 and DR-005). Every statistic is computed by code that is tested against independent Python implementations.

The two topic labellers (keyword rules and the bring-your-own-key LLM) have a model card: intended use, data provenance, evaluation with intervals, known failure modes with examples from the gold set, and ethical considerations.