Survey Methodology Insights for State of Specialty NeedleArts 2013

Can you trust the results of TNNA’s State of Specialty NeedleArts surveys of needle arts enthusiasts and make business decisions based upon them? Yes. The survey questions are valid and reliable, the number of responses statistically significant, and the sample valid.

All of the questions on The State of Specialty Needle Arts surveys (and any other Hart surveys) have been pretested to ensure they are valid and reliable. This includes administering the questions to people one-on-one and making sure each question is easily understood; if a question is confusing we rewrite it until it is understandable.

The spending questions are intentionally top-level. People would have difficulty remembering exactly what they spent on knitting needles, for example, so spending questions are kept more general. People also get frustrated if questions are too detailed. Ideally the study would pay people to keep spending diaries or conduct monthly spending surveys, but then research costs would rocket to $250,000 or more, which would be crazy for the needlearts industry. (The cost of the TNNA study was way, way less than that.) The per needle artist spending medians and percentages per product category have been validated by the industry and by major retailers, who say it is close to what they see in their own businesses. The spending percentages per product category and spending per person have been consistent over the nine years we have been conducting the surveys.

12,000 responses to a survey (in the case of knitters) is way more than needed, but is great for looking at slices of the market. 450 responses is enough for a valid sample, assuming the sample is not biased. Given the range of sources that contributed respondents, the knitter and crocheter response sets are a valid sample of knitting and crochet enthusiasts. Remember these are enthusiasts and not just any person who picked up a knitting needle in the past year. Also, note this was an online survey available to anyone, so we did not control where responses came from. We did not track IP addresses or referring URLs, to protect anonymity. People reposted the survey links all over the place. The survey did ask people what Web sites, Facebook pages, etc. respondents used, and they were a wide range.

So, in sum, The TNNA State of Specialty NeedleArts studies use a cost-effective research methodology that generates data you can depend on for good business decisions. For more information see the study overview.

NEA Fiber Arts Participation Methodology Note

This methodology note comments on the posting regarding growth in fiber arts participation.

Hart notes there was one methodology change to the fiber arts group definition in 2012 that likely did not affect participation rates in the group: The NEA added knitting to “weaving, crocheting, quilting, needlepoint, or sewing.”  Hart analysis indicates this did not affect participation rates because the group of adults doing weaving, crocheting, quilting, needlepoint, or sewing already includes nearly all the knitters since so many knitters also do at least one of the other fiber arts. The Craft and Hobby Association Attitude & Usage Study data for the year ending 3/31/2011 shows the average fiber arts (“needle craft” in CHA terminology) participant does 1.6 different fiber arts, as calculated by Hart Business Research. For example, for knitters 48% also crocheted, 25% or more also sewed, and 21% quilted.

Are Online Surveys Accurate?

Recent Hart research indicates online surveys mostly give an accurate picture except when the questions are about information use and purchasing habits. If these two topics are of interest to your business then consider doing offline surveys or interviews.

Information Use Questions

  • Where do you get how-to information?
  • Where do you get creative inspiration?
  • Do you use Facebook?
  • Do you take workshops or seminars?

These questions yield different results depending upon whether the survey is online, printed and mailed, or done as an interview. Online respondents will of course emphasize online information and learning sources. Interestingly, both online and printed survey respondents may emphasize information use because the survey itself is a “test” of reading interest. If you need to survey people who are less reading oriented, try conducting the survey as an interview or a short and simple printed survey and offer a good participation incentive.

Purchasing habits questions

  • Where do you buy creative supplies?
  • What information sources do you look at before making a purchase?
  • What store attributes are most important to you?

Online survey respondents will report higher rates of buying online, using online purchase information, and even magazine reading about products. In contrast, in-store respondents will  be more likely to say they buy mostly in stores and don’t consult much information before making a purchase. These in-store respondents are more representative of the general population. (Side note: only about 5% of U.S. overall retail sales are online.)

So where should you survey?

Conduct a survey online if your business’s market is information-oriented people or your business is online, such as a publisher, online store, or e-learning site.

Conduct a survey in stores if you are a brick-and-mortar store or a supplier to them. This could be done as an interview or a short-and-simple printed survey to be completed in the store. If the cost of an in-store survey is a significant issue, however, you may have to survey online. If that is the case, don’t take the results at face value – Interpret them knowing the responses are higher than they should be around information use and online purchasing.

How to Make Sense of CHA Attitude and Usage Study

There are three things to keep in mind when using the Craft and Hobby Association’s (CHA’s) Attitude & Usage Study U.S. data sets from 2005 to 2011.

  1. The accuracy of the market sizes varies from +/-10% to +/-35% or more, depending upon how large or small the market is.
  2. CHA changed the names of categories (woodworking became wood crafting, etc.) and went from 30 to 47 segments in 2010, making many segments seemingly not comparable.
  3. CHA changed the survey from print to online in 2010, attracting a slightly younger and more male set of respondents.

So, what do you do if you want to compare the data from 2005 through 2011?

  • Always look at the household participation rates for each segment. Hart estimates if less than 5% of US households participate in that segment, the CHA Attitude & Usage data is much less reliable. This affects leather, mosaics, macrame, print making, paper mache, needlepoint, fiber art, needle felting, and mixed media.
  • Take a close look at the category changes. There are poor matches over time for woodworking/wood crafts, wedding crafts/wedding bridal, food crafts/cake decorating, fine arts/art and drawing. Don’t compare 2005-2009 with 2010-2011 for these.
  • Set up a spreadsheet with all the data you’re interested in and graph how it varies. Some category data resembles an EKG, so don’t bet your venture capital (or credit card limit) on those.