flexiblefullpage - default
Currently Reading

Customer Feedback: The Test Kitchen

billboard - default
Customer Satisfaction

Customer Feedback: The Test Kitchen

A lot of home builders are giving themselves heartburn by insisting on more survey responses when accurate results can be obtained from a much smaller number of returned questionnaires. Why the focus on quantity vs. quality?

By Paul Cardis December 31, 2005
Chili cook-off and tasting
Photo: chili cook-off, University of Texas at Arlington News Service Photograph Collection
This article first appeared in the PB January 2006 issue of Pro Builder.

Imagine yourself judging a chili cooking contest, with long tables lined with big steaming pots of homemade chili. Given that each pot of chili has been stirred well, you are able to taste just one spoonful to determine how the whole pot tastes. That's because the small sample accurately represents the entire pot. Now imagine if you had to eat half of the pot in order to make the same determination. There's not enough antacid to relieve that heartburn.

A lot of home builders are giving themselves heartburn by insisting on more survey responses when accurate results can be obtained from a much smaller number of returned questionnaires. Making matters worse, there are a number of research companies tying to woo builders away from their current research program by claiming you need 80 percent of surveys filled out.

Surveys: Quantity vs. Quality 

Why this overemphasis on quantity instead of quality? Besides lack of knowledge, it could be human nature. No one likes bad news, so there's a natural response not to believe it, and many people will try to explain it away. "How can this be?" they wonder. "Clearly, the survey was only filled out by our dissatisfied customers. All of our happy customers must have thrown it away." Such reactions, however, are nothing more than misleading thoughts by those grasping to make sense of results they prefer not to accept.

A quality survey administered properly will yield accurate results with less than 80 percent response rate. Best of all, the response rate needed to achieve results that are 90 percent accurate is often a lot smaller than you may think.

The key is to give your program time to reach the targeted response rate and you will have the same group as if you received all of the surveys.

If you built more than 125 homes you do not need 80 percent of the surveys returned to accurately gauge your performance in customer satisfaction. That's because the power of randomization and avoiding bias comes into play. Just like that stirred up pot of chili, all you need is a tablespoon to get proper taste, given that the pot is big enough. If the pot were too small you might scoop up a burnt edge that doesn't reflect the true quality of the chili.

Sure, the first surveys to pour in might be from angry customers anxious for an opportunity to vent their frustrations. Given time, however, the total number of returned surveys will be a proportional representation of the entire customer base. The key is to give your program time to reach the targeted response rate and you will have the same group as if you received all of the surveys.

Survey Target Response Rates

Assuming that you are surveying all of your customers, what response rate should you aim for? Generally, as the population size increases, the needed response rate goes down. In our experience, most larger builders can be reasonably confident in the overall company results with just a 20 percent response rate. Though higher response rates might give you more peace of mind, the fact is that the results don't change significantly. Once you reach 40–50 percent, you are as good as its going to get with more than 125 homes. The chart above shows the necessary number of surveys needed given the number of homes built. Assuming we want to be 90 percent confident in the results and the maximum scores could vary being only +/- 6%. (on a 6 point scale, this would mean scores could go up or down +/- 0.36; on a 10 point scale it would mean scores could go up or down +/- 0.6).

A quality survey administered properly will yield accurate results with less than 80 percent response rate.

NRS once conducted a survey for a builder that generated a 50 percent response rate with more than 400 closings in a year. The client, however, wasn't happy with the negative results and insisted that only the unhappy customers responded, despite showing him the response rate chart. Rather than face reality, he paid us to drive the response rate up to 75 percent. Now, if the builder's theory were true, we should have seen some change in the results. Yet, despite having a 75 percent response rate, the survey results stayed the same.

This home builder is not alone in its thinking. A lot of companies overemphasize the importance of huge survey responses and waste precious time chasing surveys instead of satisfaction. Look at how often sales staff or construction supers chase customers to get the survey response rates up and you can see why this has become a real problem for many home builders. Unfortunately, while a lot of home builders spend a great deal of time and energy to increase the number of survey responses, those resources are not being used to address the customer satisfaction problems identified in the survey results.



Business Improvement and Customer Satisfaction

Imagine what would happen, however, if home builders stopped chasing higher survey response rates and instead turned their attention to improving the areas of their businesses that are disappointing customers? The results won't change significantly — even if you got 100 percent participation. Meanwhile, the things that could really improve customer satisfaction and create delighted loyal customers are being ignored.

This fact hasn't been lost for the best in customer satisfaction. Pulte Homes, which is the leader in homebuyer satisfaction in the U.S, with 16 divisions ranked #1. Pulte knows the importance customer satisfaction plays in improving its performance. So, what response rate does it use for its customer satisfaction surveys? A respectable 40–50 percent. Centex Homes ranked #1 in 10 divisions uses also uses a different survey with a 40–50 percent response, John Laing Homes the #1 builder in North America in customer satisfaction, recipient of the 2005 NRS Diamond Award presented by Professional Builder receives a 50–60 percent response rate. The list of leading companies is long who understand that focusing satisfaction with customers is more important than being distracted with unnecessary survey response rates. Make sure your company isn't wasting resources chasing surveys in lieu of customer satisfaction and delight.


Response Rates
90% confidence
+/-6% error


Number of Homes Built Response Rate Needed
125 60%
225 46%
325 37%
500 27%
1000 16%
10,000 2%


Paul Cardis is CEO of NRS Corp., a leading research and consulting firm specializing in customer satisfaction for the home building industry. He can be reached at pacardis@nrscorp.com.

Related Stories

Customer Satisfaction

Strive for Perfection

Even an attempt at perfection is difficult, but if you set perfection as your company's goal and everyone involved strives for it, you'll be winning against the competition, regardless

Sales + Marketing

Customer Experience Metrics: How Builders Can Use NPS

When it comes to measuring customer experience, the Net Promoter Score is an underrated and misunderstood metric in the housing industry. Here’s how to use NPS to gauge customer loyalty and boost referral sales

Quality Matters

Builder Reputation Basics: One Make-or-Break Factor You Can't Miss

Since the first construction meeting lays the groundwork for a great relationship throughout the build, don’t sleep on the initial meeting between your site supervisor and new homeowners

boombox1 -
native1 - default
halfpage2 -

More in Category

native2 - default
halfpage1 -

Create an account

By creating an account, you agree to Pro Builder's terms of service and privacy policy.

Daily Feed Newsletter

Get Pro Builder in your inbox

Each day, Pro Builder's editors assemble the latest breaking industry news, hottest trends, and most relevant research, delivered to your inbox.

Save the stories you care about

Lorem ipsum dolor sit amet lorem ipsum dolor sit amet lorem ipsum dolor sit amet.

The bookmark icon allows you to save any story to your account to read it later
Tap it once to save, and tap it again to unsave

It looks like you’re using an ad-blocker!

Pro Builder is an advertisting supported site and we noticed you have ad-blocking enabled in your browser. There are two ways you can keep reading:

Disable your ad-blocker
Disable now
Subscribe to Pro Builder
Already a member? Sign in
Become a Member

Subscribe to Pro Builder for unlimited access

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.