I want to believe.
In January, District 113 conducted a survey of area residents to learn more about spending priorities in and Deerfield High Schools. After the survey was completed, I .
Earlier this month, District 113 announced that the survey results were in. The findings were presented at a meeting of the district's market research study group, and while that meeting's minutes are not posted, the presentation charts used are available online. Unfortunately, despite my earlier column, I didn't receive any advance notice of the plan to present the results. As such, I can only base my opinion today on what's in the charts and the District's news release; these are the same tools any interested parent or voter could obtain.
In my day job, I run a brand management team where we rely on market research and analysis for a variety of activities. As such, I am quite used to reading the results of market research. Normally, the number of pie charts and bar graphs is enough for a whole series of Dilbert comic strips, which is why I was surprised to find the District 113 report is more of a qualitative narrative. Some of the individual statistics are cited, but no raw data is ever actually provided. I should clarify: by raw data what I am seeking is the actual number of respondents who indicated each particular answer or choice to the survey questions. I'm not looking for names and email addresses.
In the discussion on my last column, it was somehow asserted that publishing raw data would be unethical. I don't see how that would be the case. All of us who responded to the survey believed we were responding to a public government entity, and would have no expectation of the aggregate data being private. Thus I was surprised when I received a follow-up email from district113confidentialresearch at gmail.com about the survey; I did not know how this random gmail account got my email address. A level of transparency, and a chance for the district to build more trust with the community, seems to have gone missing.
Without the data, we are left to believe the interpretation of the market researcher, Marci Cohen of Highland Park. I have no reason to doubt her credentials; according to District 113 and LinkedIn, she runs a market research firm.
Cohen does a reasonable job of setting up the validity of the survey results. While it still remains possible that respondents lied about their age, voting status, etc., Cohen claims the validated vs. non-validated responses follow similar patterns. I am therefore willing to put issues of sample bias aside while reviewing the results.
The survey touched on several key areas of potential investment for the schools. Unsurprisingly, survey respondents were in favor of repair and maintenance. The survey also asked whether the 1905 buildings at Highland Park High should be replaced or renovated; it is reported that most agree with renovation. However, the limited data provided and the qualitative analysis in the district presentation don't line up here -- 39 percent said that there is "a lot of heritage" to these buildings, yet the presentation claims "The heritage of the buildings does not matter that much to the community." Personally, it matters to me, and apparently four in ten of the survey respondents.
Considering that much of the referendum was targeted to improve sports and physical education facilities in the schools, survey responses on these topics are surprisingly only covered in one page of generalities. From this point in the presentation forward, none of the survey results are called out specifically, and issues such as pool or field replacement are never discussed. The subsequent topics of job training and academics are also only lightly touched.
Putting myself in the shoes of the average voter, the district's presentation of survey results therefore communicates relatively little new information. It confirms that the community does not support the full scale of the 2011 referendum, but other than the 1905 buildings, the presentation does little to further typical voter knowledge of what the relative priorities are for voters in the district. Furthermore, the lack of published data means that we cannot draw conclusions of our own, and are left to rely on one person's interpretations.
If I were District 113, trying to win in the court of public opinion, I would publish the validated data. The survey was conducted in part at district, and therefore taxpayer, expense (postcards were mailed to district residents). Don't we deserve the ability to draw our own educated conclusions?
I disagree with your comment that the data and qualitative analysis about the 1905 buildings don’t line up. I think Ms. Cohen was trying to say that respondents appreciate the historic nature of these buildings and would like to try to rehabilitate the buildings if it makes economic sense. But, respondents don’t agree with the notion of restoring these buildings damn the cost. At some point, people believe it is worth tearing them down. These are not inconsistent positions. You stated in a past post “It will be important for the District to prove that somehow the survey hasn't been unduly influenced by repeat voters, students, or partisans, or the survey was pointless.” From the survey results, I find no evidence of any undue influence. Do you?
I'm conflicted about the idea of releasing the data. I agree that it is good to see the raw data, but I also believe that the analysis methodology would be debated and become a sideshow. Like you state, the survey doesn't really tell us anything new, but the fact that it validates general undstanding is important. Could I debate that the raw data reflects a more active pro-referendum group than anti-referendum group? Sure, but that wouldn't be valid, as it ignores the data provided by the referendum vote last year. What I would hope is that, if the raw data were in fact released, you would be a very vocal proponent of the need for data balancing and normalization.
I'm not sure I believe it as strongly as it is being asserted here. I personally conducted the survey twice, the second time being for the previous column. The idea that the only validation was an email is still disturbing to me, as if I was not who I said I was, I could still give a well-known email address in filling out the survey a 2nd/3rd/4th etc. time and the validation wouldn't "bounce". If I was underage I could lie about my age and voting status with no fact-checking done. But normalization doesn't matter when the analysis presented is mostly qualitative. Only because the three of you bring it up, it feels like it's now the lead talking point on why this report-out is valid. See, without some of the actual data, we have no way to know if the words on the slides are indeed representative of what was said by survey respondents. That's the part that lacks transparency and thus, makes the statistical technique applied a secondary issue.
Ed - this is your second article on the subject, and you don't claim to have a) attended any meetings, or b) contacted Marci or anyone else on the committees for comments (sorry, reviewing her website and LinkedIn profile don't count ). I've at least been clear that I feel those are the least of the things you should do before presenting a countering argument or critique. How much weight would you give to a critique of Notes based on only ever seeing a PowerPoint presentation, and not actually using the product (or maybe talking to the product manager)?
In addition, I note that whenever I check the results of one of these surveys, it appears that I can vote again. When informed of this, the Patch editor told me that he hope respondents won't try to work around the system and will vote only once. Ed has mentioned that the District 113 survey was difficult, if not impossible, to validate. I would think it's easy to state whatever demographic information is requested, such as age, sex, etc., on any online survey. So i guess a teenager who is too young to vote and isn't a taxpayer might take the survey and maybe more than once if they have the time. Teenagers have been known to do much worse on the Internet, but, of course, not a Highland Park teenager.
What is preventing you from joining the meetings and getting access to the raw data that you supposedly so desperately want? Maybe even ask questions of a few committee members who seem very open to explaining their process? Instead, you just fire away with the same influential purpose you're trying to speak out against in the survey. I believe the meetings are once a month. If you can't find time once a month, then you must not want the information all that much. I am SOOOOOOO tired of people who obviously have another agenda bashing this process when it is WIDE open to the public, and they don't take advantage. Yes, Ed, we can see this coming a mile away. You won't like what the board and committees present, and you're building your case. We get it. Now get off your keyboard and go get the information you're looking for and quit pretending that it's not there for anyone who REALLY wants it. And lastly, how much more arrogant can you be to believe that YOU should have been given "advance notice of the plan to present the results?" If you want to be little junior journalist, do a little reporting ... anything. (As an aside to the Patch, the same column is posted at another URL (http://deerfield.patch.com/articles/what-to-make-of-district-113s-survey-results#comments) for the Deerfield paper. Excuse my double posting, but it looks like this is where the conversation is happening.)
Did you ever consider the reason for the "double posting" is that the Village of Deerfield is part of School District 113? I would imagine property owners in Deerfield have a sake in the situation at HP High School. So I wonder: Is this only a Highland Park issue or one for all residents within SD 113?
My apologies for communicating poorly. Neither the message to the Patch nor the reply to you were meant as shots at anybody except myself.