Survey Says...

What's wrong with the way District 113 is seeking community input?

Over the last few weeks, District 113 conducted a survey that solicits community input about the future of our high schools

The survey questions covered roughly two broad areas: a post-mortem on the , and areas of need for the 113 schools. While I expect the results of the survey to start appearing in district communications soon, I am concerned about the way it was conducted and also the content of the survey, worries shared by many other District 113 residents.

The District 113 survey was widely publicized. District residents received a postcard soliciting survey responses, news stories provided links, and emails went home to booster club parents. My wife first heard about the survey from a city commissioner; within hours, both of us had taken the survey. That's when my spidey sense went off. 

In my day job, I run a product management organization and am quite familiar with market research survey design and output. Several things about the District 113 survey immediately worried me.

First, the survey has no validation. Anyone can take the survey unlimited times, which can lead to sample bias. The lack of validation also means survey responses could be completed by people who are not stakeholders in District 113, or students who aren't voting age. Few of the questions in the survey required answers, so you could go through and selectively provide input. Thus, there simply is no way to know how many times the ballot box has been stuffed, and whether the stuffers were eligible to do so at all. I asked District 113 about this concern. Their reply was that "Incorporating additional methodologies [for validation] would be very costly and in the opinion of the market research study group not worth the additional expense."

My second concern was the way in which some of the questions were asked. Who wouldn't indicate that they are in favor of "trustworthy elected officials" or "safe facilities" or "Adequate air ventilation"? Who would want to say that "enough quality lighting" is unimportant? Question seven asks readers to trade off these obviously-necessary components of any school. It seems like most answers will say that every possible aspect of a "quality high school" is needed; there is no weighting or reason for anyone filling out the survey to do anything other than indicate that every desire is important.

Third, I was anxious about the tone of some of the questions, and of District 113's response to criticism of the survey. The survey asked respondents to agree or disagree with statements like "I did not trust the intentions/expertise of the administrators who developed the plan"and "I was skeptical as to how well thought out the plan was." These questions wouldn't be asked in typical market research studies, because they assign a human, emotional element tied to people rather than process. Relating to these statements will not help District 113 design the next referendum, they only serve to assuage or impact the egos of those involved.

District 113, as is true in the survey itself, by asserting the qualifications of the volunteers assisting in this process. As with some of the survey questions, the tone of their response is "we know what we're doing, trust us."

This topic is too important to go with that approach. Simple techniques like IP address capture and cross-checking, use of authenticated Internet services for age verification, or serial number assignment are inexpensive and would have avoided any questions of propriety. 

I've personally come a long way over the last year on the issue of the referendum and the need to invest in both District 113 and District 112 schools. Like many voters, I am anxiously awaiting a new plan from District 113 on what the real needs are for our two area high schools. I've been persuaded by current and former families that the time is now to embark on some of the key improvements already identified. 

What I don't appreciate is a "lies, damn lies, and statistics" approach to decision-making. The district can assert that they will normalize the data from the survey, but without proper upfront controls, it is potentially garbage in, garbage out. Once District 113 starts using these statistics to influence the debate, how the numbers were arrived at will no longer be part of the discussion. It will be important for the District to prove that somehow the survey hasn't been unduly influenced by repeat voters, students, or partisans, or the survey was pointless.

Read District 113 Superintendant George Fornero's response to survey criticism .

For more news and updates from Highland Park Patch, "like" us on Facebook or follow us on Twitter.

David Greenberg January 26, 2012 at 09:13 PM
Trust is transitive. If I trust you, then by definition, I trust everyone you trust. I don't know everyone you know, so I'm unable to inherently trust you. This is an old axiom from the study of security in Computer Science. Now when you apply it to humans - some of that axiom rings true, but we also bring into play our gut feelings, and past experiences with the person. Has A.B. been a stand-up, honorable person? Has A.B. ever lied to me before? Why would A.B. want to lie to me now? and so on. If A.B. has lied to you before, you may be skeptical and want more proof of what A.B. is saying. In this case, there was a lot of apparent mistrust on both sides of the aisle during the referendum, and it's going to take time to get past it. Complete transparency in all the dealings is a good start. I give D113 kudos for the existing Study Group process - it's a lot of work, and everyone - not just the Study Group members are welcome to come. The minutes accurately reflect what we discuss (thanks Trip!) - everyone can see what's going on and contribute. So why not continue the trend and publish the raw data?
Tony Horwitz January 26, 2012 at 10:02 PM
Dave, I believe you futher my point. Distrust is transitive also. The other question might be: Who has given you such a reason to be distrustful of so many by mere association? Or do you really mean that if any single individual associated with this process has given you reason to distrust them then we are all suspect? I simply come from a place where trust is generally understood until overtly broken, and you seem to be coming from the "prove I can trust you" school. Neither is right or wrong but mine tends to work better in groups. I don't personally see how it would be harmful to make data available so you, as qualified as you may be to independently analyze it, can form your own opinion. The problem is that then any yahoo (present company excepted) can proclaim themself an expert and spout off like it represents more than just their opinion. If you are present at the market research committee when these results are discussed, I'm sure your opinion will be heard. Allowing just anyone to take pot-shots at the details is poison to the process, in my opinion. Also, I believe releasing raw data would be unethical under some code these folks adhere to.
Bringin' Down Briarwood January 26, 2012 at 10:49 PM
So let me get this straight. Leading questions with no front-end validation in a survey ... that's bad. But there's no problem with a newspaper article on a lightening hot topic that was poorly researched? In fact, it's implied that D113 should have actually made more of an effort to communicate to the author?!?!? Quick, where's Ed's silver platter? I really don't want to hear the excuse how people don't have time for these meetings. Judging from the minutes, it requires one night a month for people to offer their brilliance. If you don't have that, either: a) Shut up and let the people do their work, or; b) Do the proper homework they teach in any high school journalism class. These opinions based on minimal knowledge are maddening and irresponsible when the whole process is ABSOLUTELY WIDE OPEN. Thanks to all the respondents who are generous enough to provide each and every detail of the process. Don't forget to report your bathroom breaks also.
Ed Brill January 26, 2012 at 10:50 PM
Tony, I find this comment and others today about how the data is being collected quite concerning. So some private firm is collecting the data and not providing it in raw form to the district? Does that mean someone could potentially bias the analysis and nobody will know since we collectively - including the district - will only have access to the report? Why wasn't this disclosed as part of the survey? I did not consent to give data on how i voted to a private firm.
Ed Brill January 26, 2012 at 10:52 PM
It would be much easier to take "Bringin' Down Briarwood" seriously if you posted under your real name - so ironic given the criticisms you level at me here. I wrote this column reflective of the level of involvement that the vast majority of District 113 voters will have. If I was turned off by this survey, I shouldn't have had to go to meetings to be turned on. You want to win a referendum? You don't do it by criticizing those who have different priorities but equal votes.
Richard H Heineman Jr January 26, 2012 at 11:06 PM
Speaking of cooling things off, we need to remember that there is currently no referendum on the table. The board has not made a decision to put forth a referendum, though it is implied by the work that is being done. If and when a referendum is brought to the voters the objective is to have the present and future students and the community win not a referendum.
Bringin' Down Briarwood January 26, 2012 at 11:19 PM
I have ABSOLUTELY no problem with people who have different opinions. Zero. None. Nada. i hope they are fighting tooth and nail on the committees. It makes the process better. I have a HUGE problem with people who show a strong lack of respect for the work being done by the committees in an obviously influential manner with an opinion that is poorly informed, misleading (not in your case, Ed) and lazy when they do not care to take the time to join a process that has been made COMPLETELY WIDE OPEN to them. As for the irony of my criticisms, I'm perfectly comfortable labeling myself as a loudmouth, Internet idiot. I'm not trying to pass myself off as a legitimate journalist/columnist. If you're concerned with being taken seriously, then you might want to do your homework.
Walter (Tripp) Hainsfurther January 26, 2012 at 11:26 PM
Actually Ed makes some excellent points here. The vast majority of residents AREN't fiollowing the process, and that is a shame. What is the larger problem is that there has been no objective reporting on what is going on. Information is either coming from the District or opponents of the original referendum, whose primary complaint has seemed to be the past care of the buildings (which has been poor and they deserve to know that it won't continue) and that the District is possibly coming forward with a referendum for what everyone on the 1914 Building group believes will be a very different plan, but may not match what the Ed 1st people envision. Where is the Patch and Pioneer Press in all this?
Tony Horwitz January 27, 2012 at 12:51 AM
Ed, what I find concerning is your implication that anyone volunteering their professional expertise is anything but ethical. The marketing group (company) guards the data and will use it for the singular purpose of gauging community sentiment, by use of statistical analysis. Your information is just a bunch of numbers, like everyone else's. If you gave your email address you may be contacted (only if you wish ) to answer further survey questions and for no other reason. Every time you take a survey you give your opinion (vote) to a private firm. How would you LIKE it done Ed? Again.....the people who took the survey did so because they were interested in making their opinions known. It was too long and irritating to do more than a couple of times and there are controls for this sort of thing. Again, boils down to a seemingly deep level of distrust beneath and bubbling to the surface. I'd really like to know what makes some of us so distrustful out of the gate. See my post to David Greenberg.
Bringin' Down Briarwood January 27, 2012 at 12:52 AM
You make a good point, Walter, and it goes far beyond this column. I assume most people are pretty poorly educated about this whole thing. I know one side is abusing the h*ll out of that and using it to their advantage. I'll be cynical and assume the other side is beating the facts to death also. The Patch and Pioneer could really help this process by looking at this with a little more critical eye. As I've said in other posts, both publications would do the community a world of good by actively working to get both sides of the issue - or in this case, the sub points to the issue. Don't let these groups dictate the discussion. Please help us ask them the tough questions. A little digging and stronger look at the groups who are trying to control the message in this thing might do a world of good. E1st has put themselves front and center on this thing. Let's hear more about them. The same could probably go for the school board. As an aside: how about weekly highlighting one of the committees? I'm sure budgets are a challenge to this idea. But is there anything in this community that is more important and draws more attention?
Bob Levi January 27, 2012 at 01:56 PM
All the talk above about the "professional" market research people on the committee and the money saved by not paying for the survey brought to mind the introduction of a new product years ago. Coca-Cola decided to bring out "New Coke" some 25 years ago. Having done some consulting work for the sugar industry several years earlier, my guess was that the Coca-Cola people wanted to save money by reformulating their flagship product by using cheaper high-fructose corn syrup rather than sugar. I'm sure the folks at Coca-Cola spent big bucks having market research done before they introduced the New Coke product. Once it hit the market is flopped. Other examples of poor market research are out there, but the industry tends to gloss over those facts and only talks about successes. With no disrespect for the committee and their efforts, I can't see what the benefits might be from the attempts to get community input on the issues. I"m sure others who have followed the comments above feel the same way but haven't given any input.
David Greenberg January 27, 2012 at 02:46 PM
Tony: Part of my background is in physical and digital security, so that comes into play with my thoughts on trust. You are correct though I do come from the school of "prove I can trust you". But as I said above, when dealing with people, some of that 'proof' comes from gut feelings or from information provided by others I trust, so it's worked for me. But to each his own... As for the data, I have quite a bit of experience in statistical analysis and besides verification of the analysis, I want to see what other relationships may exist. Maybe some will be worth bringing up for discussion, maybe not. But from what's been bandied about on this site - there's over 1500 data points, I doubt many persons will want to review the data in the first place. I don't agree that it's unethical to release the raw data - the survey takers knew they were responding to a survey promulgated by a public entity (D113). I agree that data such as email addresses should be partially redacted - take the left side of the @ out, leave the remainder in. Take out phone numbers. If the comment was signed, take out the signature but leave the comment (the comment sometimes helps to explain why someone answered the way they did). Plenty of examples of this are all over government - the FCC for example publishes comments submitted, they redact the left side of the email address, but leave the names. In our case, our community is small enough that we'd want to redact the names for privacy.
David Greenberg January 27, 2012 at 02:51 PM
Where are the industry ethics promulgated? Where can they be read? I could see this in certain instruments - perhaps medically-related, but otherwise it seems strange to me.
David Greenberg January 27, 2012 at 02:58 PM
The best part about the New Coke fiasco was the public, on TV, statement by Coke's then-CEO when responding to claims that they forced the issue on purpose to boost sales - "We're not that smart, and we're not that stupid." Regular Coke came back, and that 'new coke' junk stuck around for a while longer (probably until the tanker cars filled with it ran dry ;-> ) But I digress, discussing other points of view brought out by members of the Community takes a while, it can be mentally and physically exhausting, but doing so helps to calm the collective consciousness. Rather than saying "Well, such and such wasn't discussed - I wonder what's being hidden? Is there some agenda?", one can say "Well, we hashed that out - I may not agree, but I know where they're coming from.... and what they're trying to accomplish." The two thoughts are miles apart. We're talking about a lot of taxpayer money - it's worth hashing out and getting it right - whatever we, as a community, come up with is going to be around for a while...
Walter (Tripp) Hainsfurther January 27, 2012 at 03:03 PM
With all due respect, Bob, the District has been placed in a "no win" situation here. They were absolutely hammered for spending money on the architect's planning efforts for the last proposal. They are faced with complaints on a regular basis about how they allocate their resources, such as with building maintence. They is ongoing complaints about the taxes in town. So here they have access to a group of people who do this kind of work on a daily basis for significant income, and are willing to do it for nothing. They take that route and they are criticized again. Help them out here, because the community can't have it both ways. Dr. Daryl Lund, the first superintendent of District 112 once told me (after a pro bono study I lead was dismissed out of hand before it was even released) "This community thinks that there is a direct relationship between what they pay and the worth of the product. Something that is donated has no value." How true.
Ken Robertson January 27, 2012 at 03:30 PM
Bob - your perspective lends support to exactly why community input is so valuable. Your anecdotes arrive at conclusions supported solely by your own opinions, and assumptions ("my guess" and "I'm sure") that others must agree, although you have no support for that. There has been a tremendous amount of analysis done on the "New Coke" fiasco, especially in the area of marketing and market research. You point out that New Coke was 25 years ago (it was actually 1985), and it's been a case study for business schools ever since (I studied it in graduate marketing courses in 1995). Do you think nobody learned anything from that? FYI - most analysis points to the over-reliance on the surveys, and ignoring the emotional response in focus groups, which more accurately reflected what happened after the product launch. Your sugar industry experience argument is pure conjecture (as you say "my guess") on your part, and further, more extensive research provides better insight as to reality. To simply throw out that there is a well-known marketing failure, thus we should not even try, is ludicrous.
Bringin' Down Briarwood January 27, 2012 at 03:46 PM
And that's 75% of the problem, isn't it, Walter? You and I both know that there's a strong contingent of people who simply don't want to pay the taxes. Meanwhile, they hide it with various, changing levels of rhetoric. However, the sure giveaway to their true intentions is that they never step up to the plate with an idea, real solution or attendance at one of the bizillion meetings. Bob, in a way, I agree with your comment, "I can't see what the benefits might be from the attempts to get community input on the issues." The lengths the board has gone to get input from the community are ridiculous. They should be applauded but it's a shame they needed to go to these lengths. There's sooooooooooo many people involved at this point and so many ways that expertise and even cheap opinions are being gathered that it will be impossible for any stone to be unturned. It's one of the reasons I hate all these cheap attempt to disrupt the process. The process was created for those who whined so loudly, yet most of them are not taking advantage. One side note: David, I'm not a fan by a long shot. There's several thing I wish you did differently. However, I can't cheer and applaud loudly enough that you have been one of the few more vocal opponents who have gotten in there and joined several of the committees. I probably don't agree with many of your views, but I have no doubt you're in there fighting and the final plans will be better with your input. Thanks.
Richard H Heineman Jr January 27, 2012 at 04:05 PM
The question of industry ethics applies to this survey in several ways. First it would be unethical to in any way publicly give out person specific information. Even if you do not state it the citizen assumes that this is the case. When the email addresses were collected it was specified that they would be used for follow up surveys. It would be unethical to use them in any other way. For example, groups either for or against a referendum or even the district would not be able to use this as a mailing list EVER. It is fine to publish any comments made as long as they do not specifically identify the person. If for example, someone put their name in a comment that would not be made available. Basically if the respondent have a reasonable expectation of privacy then you must meet those expectations even if you do not explicitly state it.
David Greenberg January 27, 2012 at 07:37 PM
BDB: Thank you. I try to learn from all sides, and going through the process with the Study Group - I've learned an immense amount. I'm looking forward to learning more. One of the wonderful things about the North Shore is that people aren't afraid to voice their opinions :-) We may not always agree with each other, but ultimately, at the end of the day most people can agree to disagree and still have a nice corned beef over at OUAB together...
David Greenberg January 27, 2012 at 07:38 PM
I agree - I'd never want to see the email addresses released, or the name of the person in the comments. That was made clear when the survey was taken, it shouldn't change...
RonnieTheLimoDriver January 27, 2012 at 11:53 PM
I agree with David. In issues related to government and other areas of society funded by taxpayer dollars, the more transparency, the better. EVERYONE has their own motives. That’s just life. And its very easy to use statistics to support your own personal viewpoint. I have absolute confidence in the people that have volunteered their time to this project. However, others might not be so trusting and I see no problem sharing gobs of anonymous data with anyone who would like to review it. It could probably be obtained with a Freedom of Information filing but no reason to jump to that. The best way to silent critics is to be open. I will also say that I believe the committee did make a mistake in not adding some attempt to validate the submitters. It would have been simple to log the IP address of all submitters, and time of submission. That way if you had 100 responses from the same IP in an hour, you could throw it out. That wouldn’t have cost anything, and many of the online survey services include this functionality. You could have also serialized the post cards easily with essentially no further cost. Ive done both of these things before in my own business ventures. Sometimes its not about reality; its about perception.
Bob Levi January 28, 2012 at 02:31 PM
Heres' my response to Ken's post at 9:30 am yesterday: Looks like you picked a couple of nits, Ken. I mentioned the New Coke "fiasco" happening "sone 25 years ago." Note the word "some." That was meant to imply "about" 25 years ago. It appears that comment wasn't precise enough for you. I didn't do the math and stated the rollout happened exactly 27 years ago. However, had I written the comment this past December, the event would have happened 26 years ago. I don't know the exact month, but maybe that might have helped me to be more precise. Also you mentioned my using the term "I guess" when refering to Coca-Cola switching to HFCS. I didn't have any basis for the statement so the term "I guess" implied that perhaps Coca-Cola may have take that route. Actually, it was an educated guess based on my conulting work for the sugar industry jwhen HFCS had been introduced in 1969.. The point is that it appears, Ken, (IMHO) that you and others are spending an awful lot of time trying to do something that may or may not result in anything significant. I recall HP's "deer problem" was discuss for 9 years. Due to pressure from outside the community, the City Council at the time wasted taxpayers' dollars by bring in a team to sterilize the deer. Time has shown there was little or no impact to the size of "our" deer herb because of the sterilization effort. fBTW - Many of my conservation friends thought that particular staregy was the wrong one so it's just not my opinion.
Bob Levi January 28, 2012 at 02:33 PM
Some additional thoughts: Does HP need to be so politically correct that everyone needs to be heard? The correct results may or may not happen. Just looking at some of our legislative bodies indicates that bad decisions can be made after lengthy discussion and r"research." I'm reminded of my experiences when I headed up the U.S. office of a medium-sized Japanese consulting firm. I learned quite a bit about Japan and its buseiness culture. The key is consensus thought. I know of situatations where Japanese companies made bad acquisitions after extensive discussion and resarch. No one woould accept the responsibiltiy for the poor descisions made. Is this committee headed in the same direction? BTW - I've spent the better part of my career doing business research and analysis and making recommendation to clients based on facts. I've also had my own consulting practice. A client once asked if I considered myself a good conslutant. Without sounding flippant, I responded, "I guess I am since I've always been paidl." Ask the professional people on your market research committtee the same question. It would be intersting to hear their responses, particularly if they'd be candid with their answers.
Ken Robertson January 28, 2012 at 04:09 PM
I'm sorry, Bob, you obviously missed the main points of my response to you. I don't care if it was "exactly" 25 years ago (just pointing out the precise date as background for the discussion). Now, of course, you point out that your "guess" as to the motives of Coke were formulated (pun intended) on consulting work you did in 1969, "some 15 years" prior to the introduction of New Coke. Yet, you use your "guess" as the basis for your argument. You've argued that the survey will not result in "anything significant" on other threads based on a) an E85 tank at Wolters field, b) global warming is a hoax, and now c) New Coke was a fiasco, d) deer sterilization, e) the Japanese can make business mistakes, and f) the fact that you got paid to be a consultant. I respect your right to offer an opinion, but absolutely none of these discussions are pertinent to the specific issue of school improvements other than to point out that you feel bad decisions can be made. If we all went through life with that assumption, then nothing would get done! What you seem to be pointing out is that we should just look at some facts, analyze them, then put forward a proposal. Does that sound familiar? It happened last year, and the community voted it down for various reasons. Did Coca-Cola not learn from their mistakes? Their market share was around 25% when New Coke came out, and is around 42% today. But I'm sure marketing and analysis had nothing to do with that.
Ken Robertson January 28, 2012 at 04:13 PM
Also, Bob, why don't YOU go to a wide-open PUBLIC meeting of the market research committee and ask them yourself if they "get paid"? That way, you can get your answers, and they can be recorded in the minutes for everyone to see. Of course, as you say, that's only if they are "candid with their answers". Great argument - they need to answer my questions, but I reserve the right to call them dishonest, even when I have absolutely no foundation to do so.
Bob Levi January 28, 2012 at 04:33 PM
Did I say anything about anyone being dishonest? If you interpret my words that way, it's hard for me to imaging how you and the committee will interpret the survey results. Just wondering. BTW - My initial consulting work for the sugar industry was in 1969; however, I did follow-up work for them over subsequent years. I also tend to track developments in an industry after I've worked for that indsutry, particularly trade associations. As far as the New Coke introduction, didn't any of the many case studies ever give the reason why Coca-Cola reformulated. Since sugar was the main component of their original formula, perhaps the switch to lower-priced HFCS was one of the incentives they had to change. I felt I didn't need "proof" to make my "guess."
Ken Robertson January 28, 2012 at 06:44 PM
Bob - you said "Ask the professional people on your market research committtee the same question. It would be intersting to hear their responses, particularly if they'd be candid with their answers.". Here is the definition of "candid" from Merriam-Webster: "free from bias, prejudice, or malice;marked by honest sincere expression;indicating or suggesting sincere honesty and absence of deception". I am of course assuming that you are not referring to photography in your statement. Now, given your statement and the actual definition of the words you use, it seems that, yes, you are reserving the right to call them dishonest or deceptive (i.e. not candid). FYI - I am not on the committee, therefore how I interpret the results will be of no consequence to them. However, based on your pre-disposed opinion on the validity from having seen 1/3 of the survey, along with your experience in the sugar industry making you an expert in the analysis of market research re:New Coke (by the way - there were many case studies, reports etc about why Coke was reformulated, and it had nothing to do with HFCS), it's hard for me to imagine how YOU will interpret the results. My "guess" is that you will see them as a basis for powering Japanese buses using New Coke, but, candidly, I have no "proof" for that.
Bob Levi January 30, 2012 at 10:48 PM
I received a Patch folloe-up posting email on Sunday evening that the following comment had been posted to this thread: Richard Heineman commented on the article Survey Says... "Coke allowed its bottlers to switch to HFCS in 1980 and everyone had already converted before “New Coke” was introduced." Yet today, Monday, the comment doesn't appear on the thread. Did someone flag it as inappropriate by accident?
Richard H Heineman Jr January 30, 2012 at 11:02 PM
I deleted it because I decided that it was not important
Walter (Tripp) Hainsfurther January 31, 2012 at 02:52 PM
For everyone's informatiion, there will be a meeting of the 1914 Buildings/PE and Athletic Facilities Study Group this evening at 7 PM. The meeting will be held at Deerfield High School and we will be discussing needs for Deerfield High School that should be accounted for in the Master Plan. Everyone is welcome to attend.


More »
Got a question? Something on your mind? Talk to your community, directly.
Note Article
Just a short thought to get the word out quickly about anything in your neighborhood.
Share something with your neighbors.What's on your mind?What's on your mind?Make an announcement, speak your mind, or sell somethingPost something