Why Nala asks thousands of small questions

One big question, or many small ones

A chat model reading a stack of transcripts does three jobs in one call. It picks what matters, judges how often it came up, and writes it up. You can’t check any of those without redoing the work, and the model keeps nothing for you to check against.

I split the work up instead. Nala never asks a model what the data says. It asks thousands of small questions, each about one passage or one pair of passages, the kind a person could check. It stores every answer, and plain code counts them. The way I put it to myself: ChatGPT gives you an opinion about your data; Nala gives you a record of it.

That split decided almost every tool below. The pipeline runs in seven steps:

Read

cut each answer into passages a person could quote.

Mask

replace names before any text leaves the server.

Judge

decide which passages are evidence, and what kind.

Pair

ask whether two people made the same point.

Group

turn those agreements into themes.

Check

confirm, person by person, who makes each point.

Word

name each theme and write one sentence about it, citing its quotes.

spaCy: cutting the transcript

Before Nala can judge anything, it has to cut a transcript into passages. Every later step judges passages, and a reader sees each one as a quote, so each has to make sense alone.

spaCy’s small English model splits each answer into sentences. Nala strips filler like “you know” and “I mean”, merges anything under six words into its neighbour, and splits anything over forty at a clause. spaCy runs on a CPU, costs nothing per call and makes the same cuts on every run. That last property saves money: Nala caches each judgment against the passage it was about, so if the cuts moved between runs, I’d pay for every judgment twice.

spaCy has limits. Its small model tagged Salesforce and Knak as people and Marketo as a place. For splitting sentences, that does no harm. For deciding which words to hide, it would.

GLiNER: finding the names

Interviews are full of names, and Nala masks them before any text goes to an outside model. GLiNER is a small model that finds whatever kinds of entity you name, with no training. It got the software names right where spaCy didn’t, at about 30 milliseconds a sentence on a CPU.

I tried it on a bigger job first. If GLiNER could tell which sentences were worth analysing, I could stop paying Jev to judge the rest. It couldn’t. Measured against Jev’s judgments, it scored between 0.51 and 0.68 on a scale where 0.5 is guessing and 1.0 is perfect. I kept it for names.

It still went wrong once in production. I’d asked it for a single label, “person”, so it gave that label to anything name-shaped: Garmin, Strava, Melbourne, Sunday. One participant asked for a Garmin feature, and Nala stored the request as being about “my [name]”. Masking meant to protect the participant had erased their evidence. I gave GLiNER competing labels (software, brand, place, date, job title), and now Nala masks a word only when “person” wins by a margin. On a set of real cases, wrong masks fell from 17 of 31 to 1 of 31, and GLiNER still caught all 13 real names.

Jev: the judge

I put Jev, a model from TypeSafe, at the centre of the pipeline for a product reason and a business reason.

The product reason. Most interview analysis is classification. Is this passage evidence or small talk? A pain or a goal? Did these two people make the same point? The classic approach trains a classifier for each question, and that needs labelled data from each customer’s domain. I had none. Jev is a decision model, what TypeSafe calls System One: you ask it a typed question (yes or no, a score, or a pick from a list) and it answers. It writes no prose, so it has no room to invent a finding.

Each answer comes with a probability, and Nala’s guardrails are thresholds on those probabilities:

• A passage counts as evidence at 0.5 or above.
• A passage can start a new theme only at 0.6 or above, and only if the person said it with some force. A calm description of someone’s job rarely qualifies.
• If Jev placed a person’s only passage in a theme below 0.8, Nala doesn’t count them.
• For each theme and each person, Nala asks Jev “does this person make this point?” A no keeps their quote on the page as related evidence and takes them out of the count. This check lifted the share of themes counting the right people, no more and no fewer, from a third to about two thirds.
• A theme where Jev can’t confirm enough people becomes a lead: you see it, and Nala leaves it out of the counts.
• If more than 5% of Jev’s answers in a run fail, Nala fails the run instead of shipping a thinner analysis without saying so.

A model that only writes prose gives you nothing to set those thresholds on.

The business reason. Jev is fast and cheap. It reads 20 passages a call, and analysing a ten-interview study cost about eight cents in Jev when I measured it. At chat-model prices, asking “does this person make this point?” for each theme and each person would be absurd. At Jev’s, I could ask it by default.

Two more properties made it fit. Nala caches each answer by its question, so a judgment holds once made and a rerun costs nothing. And each study pins the Jev version that analysed it, so an upgrade can’t change last month’s findings behind your back.

Louvain: themes from agreement

The first version of Discovery, the part of Nala that finds themes, clustered passages the standard way. It turned each passage into an embedding, a list of numbers that puts similar text close together, and ran UMAP and HDBSCAN to find the dense regions. The random seed alone moved it from finding 5 of 11 planted themes to 9 of 11. I couldn’t show a team a theme that appeared for some seeds and vanished for others.

Embeddings measure topic, and a research theme is a claim. “The app forgot my timer” and “I love the timer” sit close together and say opposite things.

Version two uses embeddings only to choose which questions to ask. For each strong passage, Nala finds its eight nearest passages from other participants and asks Jev, pair by pair, “is this the same finding?” Each yes becomes an edge in a graph. Louvain, an algorithm for finding communities in large networks, then finds the groups of passages that agree with each other more than with anything else. Two groups merge only if Jev judged every pair across them to be the same finding, so one loose link can’t join two themes.

I test changes on two sets of fictional interviews with themes I planted in them. There, version two found 75% of the planted themes, up from 64%, and 74% of its themes were real, up from 66%. It also cut 36 themes to 23. On real interviews it found about two more of the themes I’d marked by hand, at 1.6 times the Jev cost, a trade I’ll make at eight cents a study. A later review found the gain in themes found sits inside the noise between runs, so the gain I trust is in what a theme means: a group of people Jev judged to agree, pair by pair.

I also tested classic topic modelling as a cheap first filter. Only 10 to 33% of passage pairs that belonged in the same theme shared a topic, so the filter would have thrown out most of the right pairs before Jev saw them.

LLMs, in three places

For Nala’s first month I held a hard rule: no generative model anywhere. Nala named each theme with a quote from its own passages, since a quote lifted from the transcript can’t be invented.

The names failed as headings. A spoken fragment like “yeah the thing with the alerts” makes an honest label and a useless heading. On 20 test themes, 4 of those names worked as headings. Letting Jev pick the best of several candidates got about 6. A small LLM, given each theme’s ten most central passages, wrote 18 that worked. On a real study I compared the two blind and chose the LLM’s name for 12 of 12 themes, at under a tenth of a cent per study.

I changed the rule. The new one is narrower and holds up better: nothing generative judges a passage or counts a person. An LLM does three jobs:

1. It names each theme and writes one sentence about it. The sentence cites its quotes by number, and code checks each citation points to a passage in that theme.
2. It splits big themes into sub-themes. Nala runs the split three times and keeps the one the runs agree on most, since any single run could have come out differently.
3. It writes the summary at the top of the Brief, the report a team reads. Code builds and ranks a digest, picks the points it opens on and fills in every count, and the LLM puts them into sentences.

Nala pins one model, caches each call, and feeds it only passages Jev has already judged. A workspace can turn AI summaries off and keep every theme and count. The LLM comes last, after every judgment and count is fixed.

What still isn’t right

On 11 November I ran a full review of the pipeline. It found three problems:

• About one counted person in six doesn’t make the point they’re counted for.
• Two fresh runs of the same study, with nothing cached, share 75 to 82% of their themes on average. Stored answers hold, but first runs vary.
• Two counts in the product depended on which passages the LLM chose to read, which breaks my own rule. I’m fixing those now.

I can list these because of the stack. When Nala gets a theme wrong, I can find the judgment that went wrong, see its probability and measure how often it happens.

With one big question to a chat model, all I’d have is a summary that felt off.