Miscellany (pdf)

	"Reef Check 2004" a big splash at Sharp Island
	The Environmental Life Science Society
	Birdbrains in the Big Bird Race
	Rocky shore envy: observations vs. experiments in ecological research
	Kadoorie Farm & Botanic Garden - Wildlife updates & sightings

**Rocky shore envy: observations vs. experiments in ecological research(pdf)**

Most scientific research involves manipulative experiments in which the investigator assigns treatments to groups of whatever is being studied. In ecology, the treatments are things like the exclusion of predators, the addition of nutrients, or the artificial pollination of flowers. Normally the treatments are assigned randomly to each experimental unit: for example, one could flip a coin to decide if a particular plant (or vegetation plot) is to be fertilized (or cut or burned) or not. The advantage of such a randomized experiment is that we can be sure that the differences between the groups are either the result of the treatment or a result of chance, and standard statistics are very good at telling us which of these is most likely.

Randomized experiments are relatively easy to do when the relevant spatial and time scales are small, but are much more difficult when we are looking at processes that happen on very large spatial scales or over very long time periods. In such cases we are often forced to rely on observational studies or so-called "natural experiments", where we take advantage of natural variation in the factor of interest (e.g. soil fertility). These studies produce data that looks exactly the same as the data produced by randomized manipulative experiments, so it is therefore tempting to analyze and interpret it in exactly the same way. However, with observational studies - including natural experiments - the units are already in treatment groups and the investigator has no control over this. An example would be comparing plant growth on naturally nutrient-rich and nutrient-poor sites.

The problem with this approach is that the differences between groups could be the result of the treatment or of chance – as in a randomized experiment - or the result of some other confounding variable. With our plant growth example, for instance, any observed differences could be the result of other, unmeasured, ways in which naturally nutrient-rich and nutrient-poor soils differ, such as aeration or drainage. The possibility that the observed differences between groups are not the result of the variable of interest means that we cannot use observational studies alone to establish a causal connection. Our plants may grow faster on the nutrient-rich soil because it also has a better water supply. In contrast, in a randomized manipulative study we would assign the nutrient treatment at random to our plants so, even if water supply varied betweens sites, the fertilized and unfertilized plants would have an equal chance of being on a site with a good water supply. (Note, however, that confounding variables can be a problem in randomized experiments if they are an unintended consequence of the treatment: for example the increase in humidity that results from bagging flowers to exclude pollinators.)

Another alternative, which at first sight blurs the distinction between experimental and observational studies, is to make use of "unplanned experiments", i.e. manipulations carried out by people for reasons that have nothing to do with ecological research. If we want to look at the long-term impacts of rainforest fragmentation, for example, we can find fragments that have already been isolated for decades, which is a lot easier than creating new fragments and waiting for decades to see what happens. Comparisons between channelized and natural streams or polluted and unpolluted lakes are other examples of this approach. Unfortunately, such studies are no different from the observational studies discussed above unless we have good reason to assume that the "treatments" were applied randomly. In the great majority of cases this assumption is unlikely to be true. Human impacts, such as rainforest fragmentation, stream channelization, pollution and hill fires, do not occur at random, so there will almost always be confounding variables in comparisons with unaltered sites.

None of this will be news to rocky shore ecologists, for whom the random assignment of treatments is second nature. If a rocky shore ecologist holds a dinner party, the seats are positioned at random coordinates, the guests are seated randomly, and meals are then assigned to them randomly. Terrestrial ecologists, in contrast, sit with their friends and eat what they like – a hopelessly confounded design. But – seriously – if only fully replicated and randomized manipulative experiments are allowed, then terrestrial ecology would be limited to the small spatial scales (centimeters to metres) and time scales (days or weeks) that characterize most research on rocky shores. The great majority of interesting terrestrial phenomena - with spatial scales of kilometers or more and time scales of decades or centuries - would be forever beyond our reach.

The answer is not to abandon observational studies but to lower our expectations of statistics. We cannot avoid using (un)natural (non)experiments when looking at large spatial and time scales - the scales that are often most relevant to conservation problems - but we have to realize their limitations. With a fully replicated and randomized manipulative experiment, confidence in the conclusions is based largely on the results of the statistical analysis – the effect size and p-value. This can never be true for observational studies, including natural and unplanned experiments. In these cases, confidence in the conclusions depends at least as much on the additional information (usually from additional studies or the literature) that allows us to separate the effects of interest from the influence of possible confounding variables. The results will never look as neat as they would be if we simply pretended that we had done an experiment, but they will be nearer the truth. It should also be noted that, while ecologists are typically most interested in the causes of differences, in many practical applications of ecological research (e.g. conservation, forestry and fisheries) the magnitude of the difference is more important than its precise cause. Foresters, for instance, want to know where their trees will grow best, while teasing apart the various factors responsible for differences in growth has a lower priority.

I will end by touching on another issue, that of the independence or non-independence of the replicates, because it interacts with the problems discussed above. Most statistical tests require that replicates are independent of one another: that is, they require that what happens to one replicate is not influenced by what happens to the others. In practice, independence can usually be ensured in ecological experiments by separating the replicates by enough space (or, in some cases, enough time) so that they are unlikely to affect each other. Non-independence is less likely to be a problem with randomized experiments, because the spacing between replicates will be variable and so less likely to consistently bias the results in one direction. Non-independence can, however, be a huge problem with non-randomized or non-experimental studies, particularly if we either do not know how much separation is enough or – and this is very common in terrestrial ecology – adequate separation is impractical. As part of his PhD study, Kwok Hon Kai compared the bird communities in a natural secondary forest and an exotic plantation. He sampled birds at four points in each forest type, but the points in each type were inside the same forest patch and only 80 metres apart. Clearly these points are not independent and cannot be considered as true replicates. He therefore published the study without any statistical comparison between the forest types, but with additional information from other studies about the ecology of the bird species for which densities differed between forest types (Kwok & Corlett, 2000). The alternative would have been to leave this important question unstudied, since there are not enough similar forest patches in Hong Kong for truly independent replicates and, even if there were, it would be logistically impossible to visit widely separated sites the sixty or more times needed to get an adequate estimate of bird densities.

To summarize: randomized experiments with independent replicates allow you to make full use of the power of statistics to separate the effects of the treatment from chance variation. Observational studies – including natural and unplanned experiments – are more difficult to analyze, since additional information is needed to account for the effects of confounding variables. Careful sampling design and the use of multivariate methods can mitigate, but never eliminate, this problem. Known confounding variables can be measured and accounted for statistically, but situations in which all potential confounding variables are known and can be measured are rare in ecology, if they occur at all. However, observational studies are the only realistic way of investigating a whole host of interesting ecological questions, including most of those of practical importance. We need more and better observational studies in ecology, but we must not pretend that they are experiments.

Barnard, C., Gilbert, F. & MacGregor, P. (1993). Asking questions in biology. Prentice-Hall, UK.

Gotelli, N.J. & Ellison, A.M. (2004). A primer of ecological statistics. Sinauer, Sunderland, MA.

Kwok, H.K. & Corlett, R.T. (2000). The bird communities of a natural secondary forest and a Lophostemon confertus plantation in Hong Kong, South China. Forest Ecology and Management 130: 227-234.