What Open-Ended Answers Give Away

Three answer cards: the top and bottom ones hold identical lines and are joined by an equals sign; the middle one holds similar lines and is joined to the top one by a dashed approximately-equal sign

Picking an option takes one click. A script can do it, and so can a person on their fortieth entry of the day. Writing a sentence in your own words is different. It takes a moment of thought, and doing it again and again, differently each time, is real work. That makes the free-text box the hardest question on a form to fake, and the place where cutting corners shows.

An open-ended answer can tell you two things. On its own, whether anyone put thought into it. Next to everyone else's, whether someone already gave the same answer.

One answer at a time

Each open-ended answer is checked as it comes in:

  • Too short to say anything. A question can ask for a minimum number of words. Below it, the person is asked to write a little more before moving on. We keep that minimum low on purpose: a high one makes honest people pad their answers, and pushes the rest to paste something long.
  • No real words. A short answer made of keyboard mash, repeated letters, or strings that aren't words in the language the screening asked in.
  • The wrong language. A fluent answer in a language other than the one the screening asked in. An English answer to a German screening proves nothing on its own, but it is worth a second look.

In the visit's screening results, each answer carries its checks, and each check reads passed, failed or inconclusive. Inconclusive matters. An answer written in another language can't be checked for real words in the screening's language, so that check says it can't tell. It doesn't call a fluent answer junk. A check that doesn't apply shouldn't count against anyone.

Why we don't run an AI detector

Some answers are written by a chatbot. They are fluent, on topic and long enough, so they pass every check above. That's expected: those checks look for answers with no effort in them, and a chatbot's answer reads as if someone tried.

We tried the obvious fix. The AI-text detectors we tested missed most of the machine-written answers we gave them, and one leaned the wrong way, rating real answers as more machine-like than the machine's. A model built to spot gibberish failed differently. It marked a real person's answer as junk, because the question asked them to write about made-up characters, and made-up names are exactly what it was trained to punish.

A score we can't stand behind is worse than no score, so we don't put one on a single answer. Two things work better: asking a better question, and reading more than one answer at a time.

Questions that are hard to fake

The cheapest check is a well-written question.

Ask "Would you recommend this product to a friend?" and a canned answer can simply say the question back: "Yes, I would recommend this product to a friend." It's on topic, it's long enough, and it says nothing.

Ask instead for something with no right answer, which can't be prepared in advance. One of the open-ended questions Authentiq ships names five made-up superheroes and asks for a short backstory for two of them. There's nothing to look up and no statement to repeat back, so an answer written for some other question doesn't fit. And because everyone gets the same five names, the answers are easy to compare across a group.

Many answers at once

Some people take a screening more than once, each time from a different device and under a different name. Sometimes it's a few people working from one shared script. Device and network signals catch a lot of this. The answers catch more, because writing a new, believable answer every time is work, and people farming entries skip the work. They paste, or they keep one answer and change a few words.

So Authentiq compares the answers of everyone in the same screening group. A group is whatever you want compared: a survey, a campaign, an application round.

Identical and reworded

An answer someone else in the group gave word for word counts, with case and spacing ignored. So does one that says the same thing in other words.

Reworded is the hard part, because honest people write alike too. Ask a few hundred people about the last thing they bought, and two of them will have bought the same headphones and described them the same way. Some questions draw similar answers from almost everyone. A fixed bar for "similar enough" would flag those people, and the bigger the group, the more of them it would flag.

So there is no fixed bar. For each question, Authentiq looks at how alike the group's answers to it usually are, and marks a pair as reworded only when it stands well apart from that. This is also why rewording needs at least 30 answers to measure against. In a smaller group, only identical answers are matched.

Rare details

Open-ended answers aren't the only thing compared. A postcode, a date of birth or a full name that someone else in the group also gave counts too.

Common answers don't. Two people choosing the same gender tells you nothing. Two people giving the same postcode, the same birth date and the same twelve-word answer tells you a lot. Long and rare answers weigh the most; short, everyday ones weigh little.

One match, not a pile of coincidences

A person's score reflects the most they share with any one other person. A postcode shared with one stranger and a short answer shared with another don't add up: in a big group, everyone shares something with someone. What counts is sharing several things with the same person.

What you see

Open Screening → Groups in the portal. Each group shows how many people were compared and how many of them sit in a cluster. For each person you get:

  • a similarity score, with the highest at the top;
  • a cluster (C1, C2, …) when they share two or more rare answers with others, so a ring of entries reads as one cluster instead of a list of unrelated rows;
  • their closest match: which answers they share, with how many other people, and whether each one is identical or reworded;
  • how many other entries in the group belong to the same person, as Authentiq's identity resolution sees it.

For details like a postcode or an age, you see the answer the person gave. Open-ended text and full names are never shown there. Any group can be downloaded as a CSV, which holds no answers at all. A group is compared again a few minutes after each new answer arrives.

Good to know

  • Answers are only compared inside one group of one account. No other customer's answers are ever compared with yours.
  • The group comparison is a review view. It doesn't change trust verdicts, callbacks or API responses.
  • No check here claims to know who wrote an answer. They show effort, language and sameness, next to everything else Authentiq knows about the visit: the device, the network, and its history across visits.

Turning it on

The per-answer checks come with the open-ended questions: put one in a screening, and its checks run on every answer. In the Authentiq docs, Hosted Screening Integration → Screening Checks lists them and what each result means.

To compare answers across a group, give each screening a group when you create it. Hosted Screening Integration → Comparing answers across a group covers the setup, the portal page and the CSV.