[!NOTE] Manuscript in Preparation Joint work with Robert H. Tai. The interview transcripts are private, so no code or data are linked.

Overview

Open codes arrive as free text, so their recurrence across repeated runs moves with the threshold at which two labels count as one code. Deductive coding avoids this because each iteration returns a verdict on a code fixed in advance, as in the large language model quotient (LLMQ) of Tai et al. (2024), the share of iterations marking that code present. This study extends LLMQ to inductive coding and measures how far that figure changes when the matching rule changes.

A local Llama-3.1-8B-Instruct model coded 742 excerpts from 30 semi-structured interviews with university alumni in up to 20 independent iterations per excerpt. Labels were embedded with nomic-embed-text-v1.5 and matched by cosine similarity, and the pairwise match rate between iterations of one excerpt was compared with the rate between excerpts from different participants as chance.

Findings

0.899 was the pairwise match rate between iterations at a cosine threshold of 0.5, or 0.867 after correction for chance. The same runs gave 0.196 at 0.875, the threshold used to build themes, and every one of the 30 participants stayed above the other-participant rate at every threshold tested. Mean theme LLMQ was 0.193 at cosine 0.875, ranging from 0.18 to 0.22 across clustering thresholds from 0.82 to 0.92.

We find that five local models converged above chance, with themes carrying between models at cosine 0.75 at about half the rate they recurred between split halves of one model’s runs. Theme inventories from an unprimed and a minimally scaffolded prompt differed, but the difference could not be separated from run-to-run resampling under any of three null models. Because no human judged whether matched labels mean the same thing, the figures support no claim about validity, and recurrence is reported as a curve across thresholds with chance, the embedder configuration and the iteration count.

Pipeline

Transcripts are split on speaker turns into question and answer excerpts. Within each excerpt, labels from the iterations are grouped by average-linkage clustering, and a theme’s LLMQ is the share of iterations that produced it. Corpus-wide codes are clustered into 30 sub-themes and 12 top-themes, with the model used only to name and define each cluster, and a recovery step re-anchors each evidence quote to verbatim transcript text, recovering about 96 percent of quotes under both prompt framings.