Experiments and observational studies

There are two main ways to study cause and effect.

Experiments

In an experiment, you decide who or what gets the treatment. The strongest design is the randomised controlled experiment: you assign the treatment at random, so the groups are alike in every other way on average. Then a difference in the outcome is most likely caused by the treatment.

A/B tests on websites are randomised experiments: half the visitors, chosen at random, see a new design. In machine learning, most experiments are fully controlled: you choose the data, the model and the settings, and change one thing.

Observational studies

In an observational study, you only watch. People or things choose their own "treatment", and you record what happens. Examples: comparing the marks of students who chose to use an app with those who did not; studying which hospitals use AI tools.

Observational studies are often the only option, because an experiment would be impossible, too costly or unethical. You cannot randomly make people smoke to study cancer.

The danger: confounders

A confounder is a third factor that affects both the "treatment" and the outcome. Students who choose to use a study app might also be more motivated. If they get better marks, was it the app or the motivation? In an observational study you cannot be sure.

RememberUse the word "causes" only when your design can support it. For observational results, say "is linked to" or "is associated with".