Reproducible Research › Module 5: Data and models

Datasheets for datasets

Lesson 2 of 3 · 2 min

Previous

Every electronic part comes with a datasheet: how it was made, how to use it, and where it fails. In 2018, Timnit Gebru and colleagues proposed the same for datasets. Their paper, "Datasheets for Datasets", was published in Communications of the ACM in 2021.

The questions a datasheet answers

Motivation
Example questions: Why was it created? Who funded it?
Composition
Example questions: What's in it? How many items? Is anything missing? Does it contain personal data?
Collection
Example questions: How was it collected? Who was involved? Did people consent?
Preprocessing
Example questions: What was cleaned, removed or labelled, and how?
Uses
Example questions: What is it suitable for, and what should it not be used for?
Distribution and maintenance
Example questions: Licence? Who maintains it? How are errors reported?

Why it matters

Datasets shape what models learn. Without documentation, people use data for purposes it was never fit for. Well-known face datasets, for example, were found to have far more light-skinned faces than dark-skinned ones, and systems trained on similar data performed worse on darker-skinned faces.

For your own data

You don't need a long document. A one-page datasheet answering the key questions, kept next to the data, is a huge improvement over nothing.

RememberA datasheet tells people what's in a dataset, how it was made, and what it should and shouldn't be used for.