Is the survey representative of the people on the island?
There are approximately 4,393 adults on Orcas Island. Our survey might say that 50% of the 400 people who answered a question selected option ‘x.’ If we actually went out and extracted an opinion from all 4,393 adults, then 95 out of one hundred times that we asked the full group of 4,393 people, we would find their responses to be within the margin of the red lines on the graph. We might expect about half of the time, the majority would be below the 50% line. Conversely, about 50% of the time they would be above the 50% line.
The Margin of Error is dependent on several variables. The size of the population, the size of the sample (our survey), and the level of confidence that we want to have in our results (we chose 95%).
If we wanted to measure sub-groups, such as the number of people over a certain age or by where they live, we would have to know the size of the population in that group so that we could estimate our margin of error accurately.
The formula for measuring the margin of error
The formula to calculate the margin of error where the population size and the number of respondents is known is:
The population of Orcas that is over 18 is approximately 4,393.
The population in the UGA as of 2010 is 1,078.
We know that our survey is representative of the island population’s geographic location.
If you want the excel sheet that we use to calculate our margin of error for various questions, please send an email to firstname.lastname@example.org
We used a confidence level of 95% for our standard, so z equals 1.96.
We used a value of .5 for p, which is the standard used by the industry.
Our population size varies, depending on what we are measuring, as does the number of respondents.
So, is the survey representative of the people on the island?
The answer depends on which group you want to use as the measure. There are several ways of looking at the island. We used the demographics section to attempt to help us verify that those answering the survey were typical of those on the island. There are a nearly infinite number of subsets that we can create.
For example, we could have asked about eye color then tried to make sure that the number of people answering the survey matched the eye color population. As you can see, some groupings make more sense than others as relevant to the questions being asked and answered.
We chose to ask about: age; length of stay on Orcas; where you live, work, own property; home ownership; and income-related questions.
We opted to leave the survey open, allowing anyone to take the survey. Alternatively, we could have imposed a quota that said we have too many people over the age of 66 taking the survey and no more are allowed to do so. Or, we could have taken a random collection of answers and discarded them because they didn’t match the demographic we were looking for. Or, we could use all results and note that some demographics are under- or over-represented, letting the reader of the results consider that as they use the data. We opted for the latter approach.
We asked eight demographic questions. We have island statistics for geographic distribution and can approximate age groups on the island. We can compare the responses of those taking the survey to the Census data in order to take a measure of whether our survey is representative of the community. We show that information when it exists.
Do you live in the UGA
This survey is about the Eastsound Urban Growth Area (UGA). The proposed policies all deal with the UGA. One of the things we wanted to know is whether the people who live in the UGA agree with the population that lives outside of the UGA. For that to be a representative measure, the ratio needs to be similar or we need to have a large enough population that live in the UGA for the survey to be statistically valid.
It is. The percentage of the people living in the UGA who took the survey is very close to the ratio of the UGA population to the island population. We measured this against those starting to take the survey and those who completed the survey.
There are 30 Census blocks in the Eastsound area, and the boundaries of these blocks exactly match the boundaries of the UGA. A Census block is the smallest area measured by the Census Bureau. For 2010, the Census reported the island population to be 5,211 and the UGA population to be 1,078, resulting in 20.7% of the island’s population living in the UGA. Our survey has 115 people out of the 605 who answered this question as saying they live in the UGA. This is 19.0% of the total. The ratio of those who live in the UGA to the rest of the island is very representative of the island’s population. There were 388 people who completed the survey; 20% of those who did so live within the UGA.
Is the survey representative of the island as related to age?
We don’t know how the Census counts the college age group. If the Census counts those individuals who are away to school as residents, then we would need to reduce the island population accordingly. That would cause the survey to be more representative of those who were on island at the time of the survey. The survey was conducted in October of 2017.
If the Census does not count individuals who are away to college but who claim the island as home, then our survey poorly represents the opinions of the 18 to 25 age group. An appeal was made to some high school individuals to take the survey, but almost none did.
Another way of looking at the data is to compare it with the voter rolls. We have pulled the record of who voted in the November 2017 vote. This record includes the age of the voter that voted.
Here we see that those over the age of 66 are under-represented in the survey whereas they are over-represented when we compare the survey takers to the census. In both comparisons, the 18-25 year olds are under-representative in the survey.
We could compare the opinions of those over 45 to those under 36 to each other, but we have so few below the age of 36 participating that it is not a valid statistical sample to say that the 25 people in that age group who took the survey are representative of that population.
When asking if the survey is representative, it is important to know what the measure of representative is. Are you using the census, the voter rolls, or some other measure? How reliable is the measure of the your “representative” sample?
For example, if we compare the voter record to the census we find an obvious problem with one or the other. Here we compare the record of those that voted in the November 2017 vote on Orcas Island to the census population estimate. The voter record shows the age for each voter and whether they voted or not. This allows us to see what percentage of each age group voted. You will notice that 114% of those aged 80 to 85 voted. This implies that the census significantly underestimates the population of those in this age group. So do we over or under represent the people of a certain age group? It depends on what you use as your authority on how many people there are on the island of a certain age.
Is the survey representative of those that own their home vs those that do not?
It is not. The home ownership vs renter ratio on Orcas Island is 70% owner occupied vs 30% renter occupied.
Our survey results for those that completed the survey are 89% own the home they live in vs 11% that took the survey that do not. We heavily over-sample the home owner population.
We had significant percentage of the respondents that do not live full time on the island who took the survey. However, the ratio of those that spend only part of the time on island for the home ownership remained exactly the same. 89% vs 11%.
We can look to see if the opinions of those that do own their home vs those that do not are comparable. Please review this link to see the comparison. Keep in mind that our sample of those that do not own their home that completed the survey is only 43 individuals so be careful about drawing any conclusions from such a small sample size.
Questions that deal with vacation rentals and employee housing show some differences opinion between the groups.
Why might we have such a low participation rate by non-owners? Non-owners tend to be younger. They are more mobile. They may not have a financial tie to the island. Perhaps you have a better idea of why those that rent did not want to take the survey?