Skip to article
Launch

Concierge MVP vs Wizard of Oz MVP: Which to Run, and How to Run It by Hand

TL;DR: A concierge MVP and a Wizard of Oz MVP both deliver your product by hand before you build it. The one difference is whether the customer knows. In a concierge MVP they do, so you learn what the service has to do. In a Wizard of Oz MVP they don't, so you learn how people use the product as if it were built. Five things to get right:

  1. Pick by your riskiest unknown. If you don't know which steps the job takes, run a concierge MVP. If you know the steps but not whether people will use the product, run a Wizard of Oz test.
  2. Keep it small and short. About five paying customers for a concierge MVP. About 10 to 30 users over one to three weeks for a Wizard of Oz test.
  3. Write your pass mark down first. What would make you build, change or stop, decided before the first customer arrives.
  4. Watch for the personal touch. People love being served by a founder. That warmth is not your product, and it disappears when you automate.
  5. Set an end date and keep the wizard honest. No personal data the real product wouldn't collect, no advice people act on in high-stakes matters, and a refund for anyone who paid for something you haven't built.

This post is about the two manual tests. Where they sit among proofs of concept, prototypes and other kinds of MVP, with the classic examples, is in MVP vs prototype vs proof of concept. If you are not yet sure anyone wants the product at all, run a fake door test first: it is cheaper than either test here.

Concierge MVP vs Wizard of Oz MVP at a glance

Both tests replace software with a person. They answer different questions because the customer sees different things.

Concierge MVP Wizard of Oz MVP
Does the customer know a person does the work? Yes No: it looks automated
The question it answers Which steps does the job take, and will people pay for the result? Will people use the product, as designed, when it works?
What you build Nothing. Email, a spreadsheet, calls, a shared doc The front end: a page, an app screen or a form that looks finished
Who does the work You, openly You or a teammate, hidden
How many people About 5, paying if you can About 10 to 30
How long One to two weeks for a first round One to three weeks
What you measure Time per step, what they ask for, whether they pay or pay again What they do: finish the task, come back, pay
The main trap Mistaking your personal service for the product Deceiving people further than the test needs
You stop when Each new customer teaches you nothing new The result is clear or the end date arrives

In one line: a concierge MVP tests the service, a Wizard of Oz MVP tests the product.

What is a concierge MVP?

A concierge MVP is a test where you deliver the outcome of your future product yourself, by hand, to a handful of real customers, and they know you are doing it. Kromatic's Real Startup Book describes the concierge test the way a hotel concierge works: the job is "highly customized and customer-facing", done by hand for only a few customers "because it is not cost-efficient to scale."

Nothing is built. You might send a weekly email, keep notes in a spreadsheet, take orders by text message or do the job on a call. The customer gets the result, and you get to watch every step of how it is made.

What a concierge MVP teaches you

  • Which steps the job really takes. You find out in what order things happen, where customers get confused and which step eats your time. That list becomes your first spec.
  • What customers value, in their words. They tell you what they liked about the result, what they ignored and what they asked for that you never planned.
  • Whether they will pay. Kromatic's advice is to ask for payment, even a small amount, because "paying customers give more honest feedback than non-paying ones."
  • What to automate first. The step you repeat most, the same way every time, is your first piece of software. Everything else can stay manual for now.

Where a concierge MVP misleads you

It doesn't scale: you can serve five people, not five hundred, so you learn a lot about a few and nothing about reach.

The bigger trap is what Kromatic calls the concierge halo: "Founders can mistake positive feedback on the hand-delivered service for validation of the product concept." A founder who replies within minutes, remembers your preferences and adjusts the service for you is a better experience than most software. When you automate, that layer disappears. So separate feedback about the result ("the deals were good") from feedback about you ("you were so quick to reply"), and treat praise that depends on you being there as unproven.

Concierge MVP example: Amazing Airfare

In 2015, Zachary Cohn wrote up how he started Amazing Airfare, an email and text alert service for very cheap international flights. He found the deals by checking six sites and about ten Twitter accounts, and after a Facebook post about a $270 round trip to South Africa, friends kept asking him to tell them about the next one.

So he messaged five of them: "If you paypal me $8 right now, I'll add you to a list of people I email and text." Within a few minutes he had $40. In his words: "No website, no technology, no marketing, nothing. Just an excel sheet with some names, email addresses, and phone numbers." He sent the alerts as plain emails from his personal Gmail account and texted each deal by hand.

Software came one tired step at a time. A landing page and a payment form brought him to about 25 paying subscribers. "Around subscriber 20, I got really tired of texting people by hand", so the texts moved to a short script that read phone numbers from a CSV file, and soon after, the plain emails gave way to a Mailchimp template. That is the concierge pattern at its cleanest: people paid before anything existed, and each piece of automation replaced a step he had already done many times.

A shorter case: Rent the Runway's own history says that in April 2009 its founders set up a pop-up shop on Harvard's campus "where students could try on and rent dresses on the spot", in person, before renting dresses online.

What is a Wizard of Oz MVP?

A Wizard of Oz MVP, or Wizard of Oz test, is a product that looks finished and automated to the person using it, while a person does the work behind it. Kromatic's Wizard of Oz page defines it as "a product experience that looks fully built to the user", with an operator "behind the curtain" producing each response, and adds: "The user does not know."

The name comes from the film, where the great wizard turns out to be a man behind a curtain pulling levers. The user types a request, uploads a file or asks a question, and gets an answer that seems to come from software. A person read the request and wrote the answer.

Where the name came from: two studies from 1983 and 1984

The method is older than startups. In April 1983, IBM researchers John Gould, John Conti and Todd Hovanyecz published Composing letters with a simulated listening typewriter. Speech recognition was not good enough to build a typewriter that wrote down what you said, and they wanted to know whether an imperfect one would still be worth building. So they faked it: "A typist, located in another room, listened to a participant dictate via a closed-circuit TV system and typed what was said", and each word appeared on the participant's screen.

They varied what the imaginary machine could do: a 1,000-word vocabulary, a 5,000-word vocabulary or an unlimited one, and speech with a pause after every word or normal speech. With ten people much like ordinary office workers, none of whom had dictated a letter before, every version did at least as well as writing by hand. A second round with eight IBM executives who were experienced at dictating ranked the versions, and when asked to write a letter they really needed, five of the eight picked the unlimited, normal-speech version. The researchers had learned how good a product had to be before anyone could build it.

The name came the next year. In 1984, J. F. Kelley described a design method for a calendar program people could type to in plain English and called it "the OZ paradigm". Participants were told the experimenter would be in the next room "keeping more or less of an eye on the printout of the session", and "were not told of any potential participation on the part of the experimenter in the communication loop." In the first round he played the whole program himself. He used what people typed to write a first version of the software, then ran fifteen more people with himself stepping in only when the program failed, and as the program improved "the experimenter was phased out of the communications loop."

Two lessons carry over. People believe the machine: in Kelley's words, "no subject ever seriously questioned the proposition that there was a computer acting alone on the other end of the line." And the wizard is a stage, not a permanent job: you hand work to software one piece at a time until you are no longer needed.

What a Wizard of Oz test teaches you

  • Real behavior with the product as designed. People use the actual screens, so you see whether they finish the task, come back next week or pay, not whether they like a description of it.
  • What the automation must handle. Every request the wizard answers is a record of what users actually ask for. Kromatic calls that record "your spec for the real build."
  • How good is good enough. Like the listening typewriter, make the wizard slower or more limited on purpose and see where people give up.

Where a Wizard of Oz MVP goes wrong

Deception that goes too far. The customer believes software is doing the work, and if they pay, they pay on that belief. When the Guardian reported on companies quietly using people to do bots' work in 2018, it quoted Woebot founder Alison Darcy, who said the technique was not appropriate for a mental health service like hers: "Not deceiving people is very clearly one of those ethical principles."

Personal data in the wrong hands. The same report described Expensify admitting in 2017 that people, not only its scanning technology, transcribed at least some of the receipts users uploaded, with scans posted to Amazon's Mechanical Turk. One worker wrote: "I'm looking at someone's Uber receipt with their full name, pick-up and drop-off addresses." Anything your users send, the wizard reads.

Running it too long, and burning out the operator. The Guardian also recalled that in 2016 Bloomberg highlighted people spending 12 hours a day pretending to be chatbots for the scheduling services X.ai and Clara. The job was so mind-numbing that they said they were looking forward to being replaced by bots. A wizard is fine for three weeks of a test. As a hidden way to run the business, it costs the operator, and it hides how far the software is from doing the job.

Wizard of Oz MVP example: Strella

Strella makes software that runs customer research interviews with an AI interviewer. Before any AI interviewer existed, its first product "was Priya on a Zoom call with her video off", according to a Bessemer Venture Partners case study of the company. Strella recruited participants for what appeared to be an AI-led session, and co-founder Priya Krishnan joined with her camera off, "the most robotic voice she could manage", and ran the interview herself. To copy the delay of voice AI at the time, she "waited a full five seconds after each participant finished speaking before responding."

The test was one behavior: would people talk to a slow, robotic interviewer? As the case study puts it, "The five-second pause wasn't a bug; it was a test." They would. "People are shockingly willing to talk to AI," co-founder Lydia Hylton said, and only then did the team invest in building the real thing. That is a Wizard of Oz test done well: one risky assumption about how people behave, a deliberately worse version of the product, and a clear signal before the expensive build.

Which to run: match it to your riskiest unknown

Don't pick the test you like. Write down the assumption that would sink the idea if it were wrong, and run the cheapest test that checks it.

If your riskiest unknown is Ask Run
Does anyone want this at all? Will strangers try to get it? A fake door test or a waitlist, before either test here
Will they pay before it exists? Will they put money down? A pre-sale (how to pre-sell a SaaS)
What does the job actually involve? Which steps matter, and what do they value? A concierge MVP
Will they pay for the result? Will they pay a person to do it, and again next month? A concierge MVP, charging from the first customer
Will they use the product as designed? Do they finish the task and come back, with the real screens? A Wizard of Oz MVP
How good does the hard part need to be? Where do people give up if it's slower or more limited? A Wizard of Oz MVP with the wizard held back on purpose

They also work in sequence. A common path is a waitlist to find people who want it, a concierge round with five of them to learn the steps, then a Wizard of Oz round with the real screens to see whether people use it without you visible, and only then the build.

Which test to run first: if you don't know whether people want it, start with a demand test such as a waitlist or a fake door; if you don't know which steps the job takes, run a concierge MVP, done by hand and in the open; if you know the steps but not whether people will use the product, run a Wizard of Oz MVP, where a real-looking product has a person behind it

How to run a concierge MVP in two weeks

  1. Write the promise in one sentence. "We find you international flights at half price or less and text you the moment one appears." If you can't write it, talk to more people first; our customer discovery interview questions help.
  2. Pick five people who already feel the problem. People who complain about it, have a workaround and are easy to reach. Your waitlist is the obvious place to look; how to find early adopters covers the rest. Pick people who differ from each other in some way, so you aren't learning about one friend group.
  3. Map the job as steps, and do each one by hand. List what happens from request to result. Choose a tool for each step: email, a spreadsheet, a call, a shared doc. Nothing else.
  4. Charge, and say it is hands-on. Tell them this is an early, manual version and you will do the work yourself. Ask for a real price, even a small one.
  5. Deliver, and log everything. For every customer, write down the time each step took, where they were confused, what they asked for that wasn't in the plan and what you did differently for them.
  6. Ask for the second payment. The first payment says the promise is interesting. Paying again, or renewing, says the result was worth it.
  7. Decide on day 14. Read the log against the pass mark you wrote down before you started.

What counts as a pass. Write the numbers before the first customer, for example: at least three of the five pay, at least two pay a second time, and the same steps came up for most of them, so you can write the job as a checklist. A fail: nobody pays, or every customer needed a different job. Change and rerun: they pay but ask for something you didn't plan; that request may be the real product.

When to stop and automate. Kromatic's rule: stop expanding the manual service once it "stops producing new major insights", because "a concierge test is a phase, not a permanent way to operate." A practical sign is the one Cohn hit at his twentieth subscriber: you are tired of doing one step by hand. Automate that step, and only that step, then keep going.

How to run a Wizard of Oz test in two weeks

  1. Name the one behavior you are testing. "People upload a contract and come back within a week to upload another." One behavior, the one the business depends on.
  2. Mark each part as real, faked or skipped. The screen people use is real. The hard part, the matching or the writing or the analysis, is the wizard. Anything the behavior doesn't depend on is skipped.
  3. Build the front end and nothing behind it. A form, a page or a simple app that sends each request to you: an email, a spreadsheet row or a chat message. AI app builders make this an afternoon's work.
  4. Write the wizard's rules. How fast to answer, in what voice, what never to say, and what to do with a request you can't handle. If an AI model drafts the answers and a person checks them, write down when the person must step in.
  5. Recruit 10 to 30 real users from the audience you'd sell to. Kromatic suggests "10 to 30 real users" and calls a window of "one to three weeks" typical. People who pay, or give you something costly such as time or data, give better signal than free users.
  6. Run it live, and log every request. The request, the answer, how long the wizard took and anything the wizard had to improvise.
  7. Measure the behavior, not the compliments. Did they finish the task, come back, pay, tell a colleague? Compare it with the number you wrote down.
  8. End it on the date you set, and tell people. Explain what was manual, keep any promise the wizard made and refund anyone who paid for something you have not built.

What counts as a pass. Decide before launch, for example: at least half of users complete the core task once, and at least a third come back for a second time in the second week. A fail: they try it once and never return, even though it worked. A useful middle: they come back, but only for one kind of request, which tells you exactly what to build first.

When to stop and automate. Do what Kelley did. Sort the wizard's log by how often each kind of request came up, build software for the most common one, and let the wizard handle only what the software can't. Repeat until the wizard has little left to do. Stop the test, whatever happened, on the end date, and stop early if the wizard's hours grow faster than your users.

How to run a Wizard of Oz MVP honestly

The test only works if people believe the product, so the deception is real. Keep it as small as the test needs. The first four rules below come from Kromatic's Wizard of Oz page, and together they cover what went wrong in the cases above:

  • No high-stakes advice. Don't let the wizard give advice people act on in medical, legal or money matters on the belief that software produced it.
  • No data the real product wouldn't collect. Never collect or keep data you wouldn't be allowed to collect in the shipped product. Whoever plays the wizard reads everything people send, so keep that group small, and make sure the test doesn't break what your privacy policy says.
  • Refunds for what you haven't built. If someone paid for a capability that doesn't exist yet, give the money back.
  • Tell people afterward. Debrief participants where you can, and honor any commitment the wizard made.
  • Don't claim technology you don't have. Your page can describe what the product does without saying it is AI or fully automated. If the claim is what you are testing, test it with a fake door instead.

If any of this is a problem for your product, run a concierge MVP and tell people a person does the work. You lose a little realism and keep their trust.

Common mistakes

  • Running a concierge MVP to test demand. Five friends saying yes to a founder's personal service is not demand. Test whether strangers want it first, with a waitlist or a fake door, then use the concierge round to learn the job.
  • No pass mark. Without numbers written down first, every result looks like "promising."
  • Counting the halo as product love. Praise for your speed and attention is about you. Ask what they'd miss if the result arrived from software.
  • A wizard with no end date. The test turns into the business, the operator burns out and the gap between what you promise and what software does keeps growing.
  • Testing everything at once. One behavior per Wizard of Oz test. If you fake the whole product, you can't tell which part people came back for.
  • Automating too soon, or too much at once. Automate the step you repeat most, the way you already do it, and keep the rest by hand until it repeats too.

Where LaunchList fits

Disclosure: LaunchList is our product. It is pre-launch waitlist software for founders. It does not run your concierge service or play the wizard, and it takes no payments. What it does is find and hold the people you run these tests on:

  • A page to collect people. Write one sentence about your idea and the AI drafts a waitlist page, which you can edit in the waitlist page editor and publish on a launchlist.page address. Or put the signup form on your own site. A demand test like this is the step before either manual test; our idea validation page walks through it.
  • A question that picks your first five. On every plan, the waitlist form builder can add a dropdown, a single or multiple choice, or a long-text question, such as how they solve the problem today. The people with the most painful workaround are your concierge customers.
  • Your list, out whenever you want it. On every plan, export your signups to CSV from Audience → Signups, with their answers to your form's questions, and plan your concierge round in a spreadsheet.
  • Letting people in when you build. When the automated version is ready, send early access invite codes to the people from your test first, then to the rest of the list. Each person gets a personal code by email.

The page, the form questions and the export come with the Free plan, and so do personal invites, up to a fixed number (pricing has the details). To find your first five, create a waitlist and add one question about how people solve the problem today.

Concierge MVP and Wizard of Oz FAQ

What is a concierge MVP?

A concierge MVP is a test where you deliver the result of your future product by hand, to a few real customers, and they know a person is doing the work. Instead of building software, you do the job yourself with email, a spreadsheet, phone calls or a shared document. It shows you which steps matter, what customers actually value and whether they will pay, before you decide what to automate.

What is a Wizard of Oz MVP?

A Wizard of Oz MVP, also called a Wizard of Oz test, is a product that looks finished and automated to the person using it, while a person behind the scenes does the work. The user does not know. It shows how people behave with the product as if it were built, without building the hard part. The name comes from the film, where the wizard turns out to be a man behind a curtain.

What is the difference between a concierge MVP and a Wizard of Oz MVP?

Whether the customer knows. In a concierge MVP the customer knows a person is doing the work, so you learn what the service needs to do. In a Wizard of Oz MVP the customer believes the product is automated, so you learn how people use the product itself. Run a concierge MVP when you do not yet know the steps, and a Wizard of Oz MVP when you know the steps but not whether people will use the product.

Is a Wizard of Oz test ethical?

It can be, if the deception is small and nobody is harmed by it. Do not let the person behind the curtain give advice people act on in medical, legal or money matters, do not collect personal data the finished product would not collect, refund anyone who paid for something you have not built, and tell people what happened once the test ends. If any of that is a problem, run a concierge MVP and tell people a person is doing the work.

How many customers do you need for a concierge MVP?

Five is enough to start. You are looking for patterns, not statistics: which steps repeat from customer to customer, where each one gets stuck and whether they pay. A Wizard of Oz test usually needs more people, around 10 to 30, because you are measuring behavior such as coming back or paying, and one person's habits tell you little.

When should you stop a concierge MVP and build the product?

When new customers stop teaching you anything new, when you are doing the same steps the same way every time, or when you are too busy to serve one more customer by hand. Automate the step you repeat most first, keep doing the rest by hand, and move one step at a time.

Is a Wizard of Oz test the same as a fake door test?

No. A fake door test shows people a product that does not exist and counts who tries to get in; nobody gets the product. A Wizard of Oz test lets people use what looks like the real product, and they get a real result, produced by a person. A fake door tests whether people want it; a Wizard of Oz test tests whether they use it.

Keep reading

Start building your product waitlist.

Create your page, collect your first signups, and grow from there.

Build your waitlist

No credit card needed