Text and data mining, and licensing / NewsEnough
Skip to the index of this site

This is the basis on which material is read, cited, linked and, where a licence permits it, stored.

Notice 11.6, Text and data mining, and licensingLast true on

The aggregator position has been defended for twenty years and it is a short one: facts are not copyright, attribution is given, and the link goes to the origin.

Where the position is kept

A position of this kind is ordinarily kept in a terms document, where it is a promise. Here it is kept in the storage layer, where it is a constraint.

No outlet's body text is ever stored, in any form, at any stage, transiently or otherwise.

No column, cache, log, queue payload, embedding input or model prompt holds body text whose licence does not permit it. The fetch client discards a response body inside the same function that extracts the metadata, and a test plants a sentinel sentence in a fixture page and then searches the whole test database, every cache key, every log line and every queue payload for it.

Text and data mining

The UK text and data mining exception is for non-commercial research. This is a commercial product, so the exception is not relied on and nothing here rests on it.

The commercial position rests on three separate legs, and each of them holds without the other two.

  1. 01Facts are not protected by copyright, and a fact restated in our own words carries no copy of anyone's expression.
  2. 02Grade A material is reproduced only under the Open Government Licence, a Crown licence, a Creative Commons attribution licence, or a public-domain dedication that permits it. The permitted set is checked by the database rather than by an editor.
  3. 03Everything else is read for metadata and linked, and is never stored.

The EU permits commercial text and data mining subject to a machine-readable opt-out, and that opt-out is honoured. Every register that is read, with its grade and its licence, is published in the source register.

Common Crawl

Common Crawl is Grade C. It is used for extraction only. It is never cited, it is never the sole support for a fact, and it is never stored as text.

Publisher opt-outs are honoured before a fetch rather than after one, and the opt-out list is refreshed on every crawl release. Where a publisher opts out after a fact has been extracted, that fact's Grade C support is removed, which either leaves the claim standing on its Grade A and Grade B support or drops the claim.

Two licences named

Wikipedia
Wikipedia is CC BY-SA. Its text is never reproduced, so the share-alike obligation is never triggered. Attribution is given in any case, because attribution here is a rule of the product rather than a condition of a licence.
The Conversation
The Conversation is CC BY-ND. It is link only. It is never extracted into prose, and no fact is written from it.

A register whose licence is not stated does not load, so a source that cannot be named is a source that is never read.