Is There Any Data Out There? (Part II)
In Part I of this post we went over the (limited) options that engineering and consulting firms have at their disposal for gathering market data.
In this follow-up we’ll give some background on our “Data Practicality Framework”. We’ll show how it can be used by applying it to the American Institute of Architects’ Architectural Billing Index.
#1: Where does it come from?
This is another way of saying “consider the source”. If someone has a stake in the information they’re presenting there is a good chance that they are putting a particular spin on it. We can’t even blame anyone for this. It’s a result of the human factor. Information that is entirely free of bias is not only difficult to present, it’s boring. Human beings end up unconsciously promoting things they want to be true and hiding things they don’t want to admit. The numbers don’t lie but the people presenting them might.
#2: How is it measured?
Answering this question can help us figure out if we’re looking at a parameter or a statistic. A parameter is the most pure form of data because it represents an entire population. We are not very likely to see true parameters in our everyday experience. A statistic attempts to represent a much larger data set by taking a random sample and assuming it has the same characteristics as the total population. Generally speaking bigger samples are better. Bigger samples are harder to get which is part of where big data comes in. Sometimes practicality gets in the way of collecting big samples. We wouldn’t be talking about big data if it wasn’t getting easier but some metrics will always be estimates.
#3: Who is it meant for?
We’re assuming that you aren’t using the price of corn futures to project how your AEC business is going to do next month. If you are then maybe you should answer this first. Either way, we suggest that you ask this question at some point because the data might not be as relevant as you think. We aren’t completely discounting indirect data sources but given the lack of engineer and consultant specific data it bears consideration. If your firm specializes in lighting design for greenhouses you very well might benefit from agricultural data. But if you do use indirect data it’s important to understand what makes it indirect. Will your business lag the data by a few months? Maybe you are operating in a subset of the overall market? Considering these differences will help to understand what’s really impacting your business.
#4: How can I use it?
This is where we can start to leverage the whole process. The answer to this question depends on the type of data you have and what you’ve gotten out of it. Putting some serious thought into what you can do with the data could mean the difference between having it positively affect your business or not. It’s tempting to ask this question first but saving it for last prevents us from using biased, poorly measured, unrelated data for something we shouldn’t. Let your mind wander with this one but make sure you come out of it with some concrete action items. Maybe pick a time on the calendar when you can look back and see if the suggested outcome actually happened?
Now it’s time to apply this to the ABI and see what it is/isn’t good for.
Data Practicality Framework And The ABI
#1: Where does it come from?
The ABI is generated by the AIA Economics and Market Research group. This group is part of a professional organization that is polling Architect’s on a monthly basis to see what is happening with their business. In this context there’s really nothing objectionable about it. The ABI is probably as free of bias as any research can be. This is still a good 
question to ask because a “for profit” organization might be motivated to paint a more promising picture of the economy. We could go into scores of examples where editorial commentary has downright abused the use of statistics and data so the point is, always proceed with caution.
#2: How is it measured?
The ABI is a statistic and not a parameter. We know this because the AIA tells us there are approximately 700 architecture firms surveyed for this study. Earlier studies included nearly 300 respondents. So not only are we looking at a sample but we’re looking at one that has changed over time. If we compare this to the AIA estimate that 17,500 firms are owned by AIA members we see that the ABI is sampling at most, 4% of the industry (there are probably more than 17,500 firms). Statistically 700 is a great number. If the firms are randomly chosen there’s a strong chance that it actually is indicative of architect’s as a whole. But there’s one characteristic of the ABI that we know makes it non-random. The survey is only a product of those firms who respond. 
Is there some other characteristic of these responding firms that biases the data? Maybe smaller firms don’t contribute suggesting that this is more applicable to larger firms?
Another aspect of the ABI is unfortunately more disappointing. The Economics and Market Research Group isn’t secretive about this. They come out and say it right on their website. However, most people may not consider the result of this survey method. The ABI is a “diffusion index”. Survey respondents are given three questions: have billings grown more than 5%, have billings remained the same, have billings declined more than 5%. If 50% say growth and 50% say decline the score will be a 50. If 100% say stable the score will also be a 50. If 100% say growth the score will be 100. The problem we have is that the score can say 100 but the market will not have grown 100%. It has the effect of making the numbers look like a stock market index for architects, which it’s not. This is why we think it’s misleading. It’s not worth feeling cheated over because it’s not intentional but if you don’t know about it you might draw some erroneous conclusions.
#3: Who is it meant for?
This should be obvious. The ABI is meant for architects. It’s not meant for engineers or consultants. The ABI has been featured in a study published by Business Economics showing it’s relevance to construction cycles which is encouraging. It is also probably a leading indicator because architect’s have to get paid before Es and Cs do. But how much does it actually lead your business? Hard to say.
#4: How can I use it?
This is a tough one. Based on everything we’ve covered, Engineers and Consultants should see the ABI as a pulse on the status of architects and nothing more. Some businesses may not see their billings change at all in relation to the ABI. Others may see a direct correlation. What we can say with certainty is that it won’t have the same significance for everyone.
Hopefully you feel a bit more comfortable looking at new sources of data by using this framework.