What to Make of District 113's Survey Results

Ed Brill dissects the results from the high school district's survey to residents about spending priorities.

I want to believe.

In January, District 113 conducted a survey of area residents to learn more about spending priorities in and Deerfield High Schools. After the survey was completed, I .

Earlier this month, District 113 announced that the survey results were in. The findings were presented at a meeting of the district's market research study group, and while that meeting's minutes are not posted, the presentation charts used are available online. Unfortunately, despite my earlier column, I didn't receive any advance notice of the plan to present the results. As such, I can only base my opinion today on what's in the charts and the District's news release; these are the same tools any interested parent or voter could obtain.

In my day job, I run a brand management team where we rely on market research and analysis for a variety of activities. As such, I am quite used to reading the results of market research. Normally, the number of pie charts and bar graphs is enough for a whole series of Dilbert comic strips, which is why I was surprised to find the District 113 report is more of a qualitative narrative. Some of the individual statistics are cited, but no raw data is ever actually provided. I should clarify: by raw data what I am seeking is the actual number of respondents who indicated each particular answer or choice to the survey questions. I'm not looking for names and email addresses.

In the discussion on my last column, it was somehow asserted that publishing raw data would be unethical. I don't see how that would be the case. All of us who responded to the survey believed we were responding to a public government entity, and would have no expectation of the aggregate data being private. Thus I was surprised when I received a follow-up email from district113confidentialresearch at gmail.com about the survey; I did not know how this random gmail account got my email address. A level of transparency, and a chance for the district to build more trust with the community, seems to have gone missing.

Without the data, we are left to believe the interpretation of the market researcher, Marci Cohen of Highland Park. I have no reason to doubt her credentials; according to District 113 and LinkedIn, she runs a market research firm.

Cohen does a reasonable job of setting up the validity of the survey results. While it still remains possible that respondents lied about their age, voting status, etc., Cohen claims the validated vs. non-validated responses follow similar patterns. I am therefore willing to put issues of sample bias aside while reviewing the results.

The survey touched on several key areas of potential investment for the schools. Unsurprisingly, survey respondents were in favor of repair and maintenance. The survey also asked whether the 1905 buildings at Highland Park High should be replaced or renovated; it is reported that most agree with renovation. However, the limited data provided and the qualitative analysis in the district presentation don't line up here -- 39 percent said that there is "a lot of heritage" to these buildings, yet the presentation claims "The heritage of the buildings does not matter that much to the community." Personally, it matters to me, and apparently four in ten of the survey respondents.

Considering that much of the referendum was targeted to improve sports and physical education facilities in the schools, survey responses on these topics are surprisingly only covered in one page of generalities. From this point in the presentation forward, none of the survey results are called out specifically, and issues such as pool or field replacement are never discussed. The subsequent topics of job training and academics are also only lightly touched.

Putting myself in the shoes of the average voter, the district's presentation of survey results therefore communicates relatively little new information. It confirms that the community does not support the full scale of the 2011 referendum, but other than the 1905 buildings, the presentation does little to further typical voter knowledge of what the relative priorities are for voters in the district. Furthermore, the lack of published data means that we cannot draw conclusions of our own, and are left to rely on one person's interpretations. 

If I were District 113, trying to win in the court of public opinion, I would publish the validated data. The survey was conducted in part at district, and therefore taxpayer, expense (postcards were mailed to district residents). Don't we deserve the ability to draw our own educated conclusions?

Bob Levi March 29, 2012 at 01:52 PM
Ed's column has been posted for a few hours now. I'm surprised there haven't been any comments on it yet like numerous readers posted to an earlier column. Does that mean that the "research" was flawed? Does the lack of comment mean people agree with Ed? Come on, folks. Don't hold back now. The District needs your feedback.
Dan Jenks March 29, 2012 at 02:10 PM
Ed, I agree with your assessment that the survey didn’t detail much in the way of new information. I think Ms. Cohen bent over backwards to include voters who were opposed to the referendum – she specifically normed the assessment group to eliminate younger, female, pro-referendum voters who apparently disproportionately responded to the survey. I assume Ms. Cohen “balanced” her sample to reflect Census data regarding age, home ownership, etc.. I disagree with your comment that the data and qualitative analysis about the 1905 buildings don’t line up. I think Ms. Cohen was trying to say that respondents appreciate the historic nature of these buildings and would like to try to rehabilitate the buildings if it makes economic sense. But, respondents don’t agree with the notion of restoring these buildings damn the cost. At some point, people believe it is worth tearing them down. These are not inconsistent positions. You stated in a past post “It will be important for the District to prove that somehow the survey hasn't been unduly influenced by repeat voters, students, or partisans, or the survey was pointless.” From the survey results, I find no evidence of any undue influence. Do you?
Richard H Heineman Jr March 29, 2012 at 02:35 PM
The survey results were “balanced" to reflect the demographics of the voters on the referendum, not the community and only included voters. The median age was 58 and" no" voters were given extra weight to match the actual results (57% against). Dan's comments are in my view correct even with these corrections in his assumptions. If the results were balanced to reflect the community the analysis would have shown a much more positive view of he proposal.
Dan Jenks March 29, 2012 at 03:23 PM
Richard, quick question out of curiosity - how does one access this demographic information? I understand how one can easily find out the location of voters in the referendum, but how does one find out the age distribution and other characteristics of the electorate in the 2011 election? Does Lake County publish this information?
Richard H Heineman Jr March 29, 2012 at 03:29 PM
The demographic information on voters is available from the county and other sources. This was reviewed as a part of the Market Research committees work. I personally do not have the details.
Ken Robertson March 29, 2012 at 03:45 PM
Ed - this is a mostly reasonable response, but as always I'll take some issues, just for fun..I could be nit-picky and point out that the old buildings are from 1914, not 1905, so not sure where that came from. As I mentioned last time in response to your initial article, the calendar is easily accessible via the district website, and if you were missing information, surely a quick phone call would have provided the dates/times you needed. The minutes from the 2/16 Leadership team meeting (posted 2/22) reflect that their 3/13 meeting included presentation of the survey results. I'm conflicted about the idea of releasing the data. I agree that it is good to see the raw data, but I also believe that the analysis methodology would be debated and become a sideshow. Like you state, the survey doesn't really tell us anything new, but the fact that it validates general undstanding is important. Could I debate that the raw data reflects a more active pro-referendum group than anti-referendum group? Sure, but that wouldn't be valid, as it ignores the data provided by the referendum vote last year. What I would hope is that, if the raw data were in fact released, you would be a very vocal proponent of the need for data balancing and normalization.
Ed Brill March 29, 2012 at 04:19 PM
I see that everyone has learned the concept of "data normalization". I'm not sure I believe it as strongly as it is being asserted here. I personally conducted the survey twice, the second time being for the previous column. The idea that the only validation was an email is still disturbing to me, as if I was not who I said I was, I could still give a well-known email address in filling out the survey a 2nd/3rd/4th etc. time and the validation wouldn't "bounce". If I was underage I could lie about my age and voting status with no fact-checking done. But normalization doesn't matter when the analysis presented is mostly qualitative. Only because the three of you bring it up, it feels like it's now the lead talking point on why this report-out is valid. See, without some of the actual data, we have no way to know if the words on the slides are indeed representative of what was said by survey respondents. That's the part that lacks transparency and thus, makes the statistical technique applied a secondary issue.
Ken Robertson March 29, 2012 at 06:25 PM
The problem is you're trying to ferret out particular issues as if there was some sort of massive attempt to skew the data. I don't see how the results provide any real surprises, nor are they somehow moving the process in one pre-determined direction. Yes, there may have been invalid data, but I highly doubt it was widespread. If there was a high number of responses saying "don't spend any money", or "everyone should get a brand new MacBook" then okay, but I don't see it. Ed - this is your second article on the subject, and you don't claim to have a) attended any meetings, or b) contacted Marci or anyone else on the committees for comments (sorry, reviewing her website and LinkedIn profile don't count ). I've at least been clear that I feel those are the least of the things you should do before presenting a countering argument or critique. How much weight would you give to a critique of Notes based on only ever seeing a PowerPoint presentation, and not actually using the product (or maybe talking to the product manager)?
Ed Brill March 29, 2012 at 09:04 PM
Hi Ken - it's fair to encourage me to find ways to engage at a deeper level; I feel like I've taken a position similar to the level that 80% of voters will take on this issue. I reached out to the District before I wrote the first column, and had hoped that would be sufficient to start a dialog. I have been told the District regularly advises "press" (not that I personally consider myself this) about upcoming meetings; the market research committee's meeting wasn't announced and the leadership meeting date actually moved once. But I will try to get to one in the future.
Bob Levi March 30, 2012 at 12:57 PM
I'd like to weigh in regarding online surveys. Let's use the Patch recent surveys on the "best" restaurants in town as an example. There's a disclaimer stating "this is not a scientific survey." Fine, but the winner is going to get bragging rights as the best on a recent HP Patch survey. I know of at least one restaurant that asked its friends on Facebook to vote for them. They posted requests twice! Doesn't this put a restaurant who doesn't have a Facebook page at a disagvantage. In addition, I note that whenever I check the results of one of these surveys, it appears that I can vote again. When informed of this, the Patch editor told me that he hope respondents won't try to work around the system and will vote only once. Ed has mentioned that the District 113 survey was difficult, if not impossible, to validate. I would think it's easy to state whatever demographic information is requested, such as age, sex, etc., on any online survey. So i guess a teenager who is too young to vote and isn't a taxpayer might take the survey and maybe more than once if they have the time. Teenagers have been known to do much worse on the Internet, but, of course, not a Highland Park teenager.
Bringin' Down Briarwood April 01, 2012 at 06:14 PM
Unbelievable!!!! What is preventing you from joining the meetings and getting access to the raw data that you supposedly so desperately want? Maybe even ask questions of a few committee members who seem very open to explaining their process? Instead, you just fire away with the same influential purpose you're trying to speak out against in the survey. I believe the meetings are once a month. If you can't find time once a month, then you must not want the information all that much. I am SOOOOOOO tired of people who obviously have another agenda bashing this process when it is WIDE open to the public, and they don't take advantage. Yes, Ed, we can see this coming a mile away. You won't like what the board and committees present, and you're building your case. We get it. Now get off your keyboard and go get the information you're looking for and quit pretending that it's not there for anyone who REALLY wants it. And lastly, how much more arrogant can you be to believe that YOU should have been given "advance notice of the plan to present the results?" If you want to be little junior journalist, do a little reporting ... anything. (As an aside to the Patch, the same column is posted at another URL (http://deerfield.patch.com/articles/what-to-make-of-district-113s-survey-results#comments) for the Deerfield paper. Excuse my double posting, but it looks like this is where the conversation is happening.)
Bob Levi April 01, 2012 at 08:42 PM
Hey BDB, Did you ever consider the reason for the "double posting" is that the Village of Deerfield is part of School District 113? I would imagine property owners in Deerfield have a sake in the situation at HP High School. So I wonder: Is this only a Highland Park issue or one for all residents within SD 113?
Bringin' Down Briarwood April 01, 2012 at 09:29 PM
Sorry, Bob. I can understand how my cr*ppy tone could be taken the way you took it. Regarding the link, I was just trying to bring up the issue to the Patch. Usually, the Patch is pretty good about sending posts such as this one that affect both communities to one common page.
Bob Levi April 01, 2012 at 10:16 PM
BDB, I don't see where I commented on your "cr*appy tone" in your earlier post. My Mother always told me that if you can't say anything good about someone, don't say it. My comments are not meant to be personal. Sorry you may have taken the recent one that way.
Bringin' Down Briarwood April 02, 2012 at 12:50 AM
The cr*ppy tone is on me again, Bob ... both times. You didn't comment on it, and I didn't take your messages personally. I was truly trying to state that my intentions were not as angry as my initial message. Just trying to point something out. My apologies for communicating poorly. Neither the message to the Patch nor the reply to you were meant as shots at anybody except myself.


More »
Got a question? Something on your mind? Talk to your community, directly.
Note Article
Just a short thought to get the word out quickly about anything in your neighborhood.
Share something with your neighbors.What's on your mind?What's on your mind?Make an announcement, speak your mind, or sell somethingPost something