The Observatory x Dialogue AI
Internal: Script Comparison
V1 (n=53) vs V2 (n=55)
Did the Script Changes Move the Needle?
We revised the interview script based on three recommendations. Here's what changed, what didn't, and whether we can combine the datasets.
Three changes, one goal

The original script had three problems we flagged after the first 53 interviews. V2 addressed all three. Here's what changed and what it was meant to fix.

V1: Third-person only
"What are some things people often pretend to care about?"

Then moved to the next topic.
V2: Third-person, then bridge to first
"What are some things people often pretend to care about?"

"Is that something you've found yourself doing?"

V1 let cautious participants stay comfortable as observers of other people's behavior. V2 added a bridge question after each third-person prompt, pulling participants into their own experience. 53 of 55 v2 transcripts show this bridge. 69% of participants said yes and shared a personal example.

V1: Self-edit before comparison
Q11: "Did you edit yourself?"
...
Q13: "What would be harder with a human?"
V2: Comparison before self-edit
Q: "What would be harder with a human?"
Q: "What would be easier with a human?"
...
Q: "Did you edit yourself?"

V1 asked "did you edit yourself?" before the comparison questions. Participants committed to "no" and then couldn't easily contradict themselves two questions later. V2 flipped the order: comparison first, then self-editing. The comparison primes the introspection.

V1: No "easier" question
Only asked what would be harder with a human.
V2: Both directions
"What would have felt easier to say to a human interviewer instead of here?"

V2 added the inverse question. This isn't just completeness. It surfaces what AI moderation loses: warmth, body language, the feeling of being witnessed. The "easier with a human" responses are where participants articulate what they actually want from a person.

The numbers, side by side
V1
Original script, n=53
V2
Revised script, n=55
62%
Claimed no self-editing
Self-editing
claim
65%
Claimed no self-editing
45%
Said harder with a human
Harder with
human
58%
Said harder with a human
75%
Said not judged
Not
judged
80%
Said not judged
68%
Felt understood
Felt
understood
80%
Felt understood
94%
Company use acceptable
Company
acceptable
93%
Company use acceptable
13,175
Avg transcript (chars)
Transcript
length
14,371
Avg transcript (chars)
64%
High-signal transcripts
Transcript
quality
84%
High-signal transcripts
28%
Personal narratives in disclosure
Specific
stories
47%
Personal narratives in disclosure
V2 improved where it was supposed to. More personal stories (28% to 47%), longer transcripts, much higher signal quality. The self-editing claim didn't budge. The core findings held.
The bridge worked, but not the way you'd expect

The 3rd-to-1st person bridge appears in 53 of 55 v2 transcripts. When the moderator asks "Is that something you've found yourself doing?", 69% said yes and shared something. 18% deflected ("No, not me"). 13% were ambiguous.

But here's the thing: the overall self-editing claim didn't change. 62% in v1 said they didn't edit themselves. 65% in v2 said the same. The bridge didn't make people more honest about their honesty. What it did was change the texture of the data.

In v1, participants who stayed in third-person mode gave us theory: "people pretend to care about networking." In v2, those same types of participants gave us theory and then a personal example: "people pretend to care about networking... and yes, I've done it too, last week at a team event where I acted interested in a project I couldn't care less about." The bridge didn't eliminate guardedness. It created a second chance to get past it.

"I would have felt harder to say would be more of a personal example involving like my real relationships, insecurities, jealousy, moments where I've acted differently from how I felt."
V2 participant, on what would be harder with a human
"You can see their face change, even if they try to hide it. You can see the micro-expression. That's what makes it harder."
V2 participant, on why human interviews are different

The "specific personal narrative" rate nearly doubled (28% to 47%). That's real. Participants aren't disclosing more topics. They're disclosing more specifically within the same topics. Instead of "people judge others for their habits," we get "I judged my coworker for being on her phone all day and then realized I do the same thing."

Comparison before self-edit: what flipping the order did

V2 moved "What would be harder with a human?" before "Did you edit yourself?" The logic: force participants to compare before they commit to a self-report. If they just named three things that would be harder with a person, they can't easily claim they didn't edit at all.

The self-editing claim didn't move. 62% to 65%. That's noise, not signal. People still say they didn't self-edit even after naming specific things they'd hold back from a human. The resequencing didn't fix the say-do gap. It made it more visible.

But "harder with human" went up: 45% to 58%. The resequencing may be part of this, or the new "easier with human" question may have primed comparison thinking. We can't isolate the cause. What we can say: v2 participants were more likely to articulate what AI moderation gives them.

The say-do gap is not a bug we can fix with better questions. It's a structural feature of self-report data about disclosure. People genuinely don't know when their filters are active. That's the finding, not a measurement problem.
The data got dramatically better
High signal
84%
V2, up from 64%
Thin transcripts
0
V2, down from 6
Suspect
2
Same rate (4%)
Avg length
14.4k
Up from 13.2k

V2 eliminated the "thin" category entirely. V1 had 6 participants (11%) who were real people but didn't go deep enough to code meaningfully. V2 has zero. The bridge questions did this: they gave the moderator a second and third entry point into personal territory. Even participants who deflected the first bridge ("No, not me") often engaged on the second or third one.

The suspect rate stayed flat at 4% (2 per wave). One v2 suspect (Peter I.) has a 306-character transcript, which means they connected and immediately dropped. The other (MICHAEL M.) appears to be speaking a mix of Swahili and English and the AI moderator couldn't follow. These are recruitment issues, not script issues.

The practical upshot: v2 produces more usable transcripts per dollar. In v1, roughly 15% of interviews were thin or suspect. In v2, it's 4%. If you're planning to scale to 250, the v2 script will waste fewer seats.

Can we combine the datasets?

Yes. Here's why.

The core findings are identical across both waves. The absence-of-judgment mechanism works the same way. The boundary (empathy-requiring topics need a human) appears in both. The company-acceptability rate is flat (94% vs 93%). "Felt understood" went up (68% to 80%), but this is likely a function of longer, richer conversations rather than a different underlying attitude.

The script change didn't alter what people said. It altered how specifically they said it. V2 participants tell more personal stories, give more concrete examples, and produce longer transcripts. But they reach the same conclusions about the experience. Nobody in v2 discovered a new category of concern or benefit that v1 missed.

The one metric that differs meaningfully is "harder with human" (45% to 58%). This likely reflects the resequencing, not a different population. Both datasets report the same phenomenon (AI moderation removes a social barrier), but v2 participants articulate it more readily because the questions are ordered better.

Recommendation: combine. Treat all 108 interviews as one dataset. The script changes improved data quality without changing the underlying findings. For the combined write-up, use v2-style analysis (look for personal narratives, bridge-question responses) where available, and v1-style (third-person disclosure analysis) where that's what we have.

For the next 142 interviews to reach 250: use the v2 script. It produces better data at the same cost. The bridge questions eliminated the "thin" category and nearly doubled the personal narrative rate. No further script changes needed. The say-do gap is a feature of the topic, not a fixable measurement problem.

Two small suggestions for the next wave. First, the two suspect transcripts in v2 were both recruitment failures (dropout and language barrier), not script failures. If you're scaling, a tech check or language screen before the interview starts would catch these. Second, the v2 "easier with a human" question is producing interesting data about what AI moderation loses (warmth, body language, the feeling of being seen). Consider adding one follow-up probe there: "Can you describe a moment in your life where that kind of human response really mattered?" That would give the combined dataset a richer picture of the boundary.

Internal analysis. The Observatory x Dialogue AI. May 2026.