To obtain relevant texts, we retrieved abstracts from eight PubMed-listed journals that are prominent within the cancer or cell death fields; these were ‘Apoptosis’, ‘Autophagy’, ‘Cancer Cell’, ‘Cancer Research’, ‘Cell’, ‘Cell Death and Differentiation’, ‘Cell Death and Disease’, and ‘Genes and Development’. Using the PubMed Advanced Search tool, we selected abstracts that included one of these titles as a journal name but excluded ‘review’ as a publication type. The first 100 abstracts (listed by PubMed on 12 May 2021) for each journal were downloaded. Abstracts were manually processed into a total of 800 individual text files, in which only the article title and the main body of the abstract were included, with the author list and all other extraneous text being removed.

To systematically extract the key elements of experiment descriptions from texts, we focused on words and phrases defined by four main categories. These were chosen through discussion with experts in the cancer and cell death research field and capture the most relevant details of cancer and genomics experiments. The four categories were: ‘Perturbing actions’, ‘Contexts’, ‘Phenotypes’, and ‘Effects’. (i) Perturbing actions are defined experimental manipulations that relate to the regulation of any named gene or genes. We included gene names, symbols, and recognized aliases, using www.genecards.org as a reference. (ii) Contexts are the experimental models subjected to a Perturbing action, e.g. human patients, laboratory animals or cultured primary cells and cell lines. (iii) Phenotypes are the induced changes in the physical characteristics and/or behaviour of the cell or organism that result from a Perturbing action. We restricted this category to terms specific to cell death, terms specific to cancer development and to some common cell biology terms, which can relate to the behaviour of cells in both cancerous and healthy tissues. The list of terms was chosen with reference to relevant literature (Galluzzi et al. 2018) and after discussion with experts from the cell death and cancer field at Cancer Research UK Cambridge Research Institute, UK. (iv) Effects are the direction in which a Phenotype is regulated, i.e. positively or negatively. A full list of markable words and phrases is shown in Table 1.

The annotation was carried out by a biologist with 9 years of postgraduate experience in cell biology experimental research. The Multi-Annotation Environment (MAE) annotation tool was used (Stubbs 2011, Rim 2016) (http://keighrim.github.io/mae-annotation/). The framework of the annotation task was defined using a custom DTD file.

Abstracts were annotated in list order following retrieval from PubMed. Only abstracts linked to primary research articles were annotated. Reviews, commentaries, and editorial articles were excluded from the task. Annotation was carried out at whole text level; however, only the parts of abstracts containing descriptions of experiments were marked. We did not include text that formed part of the introduction or discussion, nor did we mark the text of purely descriptive studies or parts of studies, such as accounts of gene expression patterns.

Annotation was carried out at the level of whole abstracts. Words or continuous phrases were marked when they both (i) formed part of the description of an experiment and (ii) corresponded to one of the terms of interest listed in Table 2. Descriptions of experiments (though not individual entities) sometimes spanned multiple sentences. All marked Perturbing actions included a gene name or symbol, and marked Contexts included names of cell types, cell lines, species, and breeding lines when present. In some abstracts, words or phrases from all four categories were marked; in other abstracts it was only possible to mark words or phrases from some of the categories. The total number of marked entities in the corpus was 10 458.

The MAE tool allows the addition of labels, or ‘assertions’, to marked entities. Lists of assertions relating to each category of markable entity (Perturbing actions, Contexts, Phenotypes, and Effects) were defined within the DTD document and appeared as drop-down lists during the annotation process. Each marked entity was labelled with the appropriate assertion to identify its meaning. For example, ‘cell’, ‘myoblast’, and ‘keratinocyte’ were each labelled with the assertion ‘cell’, as all these terms describe cell types; ‘Rag2 -/-’ was labelled with the assertion ‘gene loss-of-function’, as ‘-/-’ denotes knockout of the Rag2 gene in both alleles. An example of marking and labelling the text of an experiment description is shown in Table 2.

Many abstracts contained multiple examples of Perturbing actions, Contexts, Phenotypes, and Effects. We created Link tags to record links between four marked entities (one from each category) that formed part of the description of the same experiment. Because the descriptions of experiments sometimes spanned more than one sentence, the four entities in each linked group were sometimes derived from more than one sentence. Where a description of one experiment generated multiple marked entities within one or more categories, we created multiple Link tags. An example of creating Link tags between marked entities is shown in Fig. 1. The total number of entities that were both marked and linked was 4697.

To measure the precision of the annotation process, a secondary annotator with cell biology experience undertook annotation of a subset of 30 abstracts (numbers 561–590). The results of the primary annotator and the secondary annotator were compared using a statistical tool within the MAE program to calculate values for Krippendorf’s Alpha-U, where a value of one indicates complete agreement. Good text marking (the correct identification of markable entities) agreement was achieved with a mean of 0.79 across all four categories (with individual values for Perturbing actions of 0.72, for Contexts of 0.82, for Phenotypes of 0.86, and for Effects of 0.79). Disagreement between annotators most often arose from the omission of whole entities rather than variation in entity boundaries, though examples of both occurred. We also achieved generally good agreement for labelling (attaching correct assertions) of marked entities within all four categories, with Krippendorf’s Alpha-U values for Perturbing actions of 0.71, for Contexts of 0.74, for Phenotypes of 0.84, and for Effects of 0.68. For both text marking and entity labelling, the greatest precision was found within the Phenotypes category.

The MAE tool XML output was converted to JSON via the python standard library. This results in the text of each abstract accompanied by lists of tags for each category. Each tag has a unique id and contains the span of text and label. There are also lists of linked entities, each with a unique id, the tag ids, and text spans for the tags. The per-tag data were converted to per-word data for each word in each abstract. Each datum contained the word, the tokenization of that word, and any tags it had. Each tag entry contained the tag id, all link ids relating to that tag, the schema label, and the tag label itself. We used the Beginning Inside, Outside (BIO) format for tagging—words at the start of a tagged span had the ‘B’ schema label for that tag with ‘I’ for subsequent tags, and an ‘O’ to denote tokens that were not tagged.

For each word, its tags were copied onto each token, and then each token was converted to an id by the tokenizer before being input into the model. Label outputs were represented as one-hot encoded vectors, with one vector for each category of label and an additional one whose elements were the tag categories themselves. For each label the ‘B’ and ‘I’ versions had separate elements in the one-hot representation. Links were represented as an upper triangular matrix with element ij (where i≤j) being one if the ith and jth tokens of the abstract were linked in any tag, and zero if not.

We used two neural network models. The first (Fig. 2) is a NER model that tags spans of text with one of the four category labels (Perturbing Action, Context, Effect, and Phenotype), as well as assigns a relevant assertion listed in Table 1. The second model (Fig. 3) links the relevant entities identified by the NER model. For example: a perturbing action can be linked to an effect, a phenotype etc., as illustrated in the example shown in Fig. 1.

Both models consist of two parts. The first is a fine-tuned pre-trained BioBERT transformer (Lee et al. 2020) and the second is a set of predictors. Each predictor is a linear classifier—a dense layer followed by a softmax non-linearity. The category predictor decides which of the four main labels to assign to the token (or predict a non-assignment); likewise; for each of the categories there is a specific head to predict an assertion for that category. The heads were applied to each of the token embeddings produced by the transformer to predict the label for that token. We used cross-entropy loss function for training.

It was found that the inclusion of a category predictor improved training by increasing the effective amount of training data for the transformer.

The linking model has two predictors, one for whether two tokens are part of tags, which are linked, and one for whether or not they are part of the same tag. Since we are using a BIO schema for our tags, this latter predictor is not required since this information is encoded into the schema. Thus, results on its performance will not be reported. However, it was found that the inclusion of this second predictor aided the fine-tuning of the transformer, thus improving the accuracy of the first predictor, by increasing the effective amount of training data for the transformer fine-tuning. During the training process we concatenated every possible pair of token embedding vectors. The model predicts if tokens were linked via binary classification. We trained the model using a binary cross-entropy objective.

For both models, we allowed the transformer model’s parameters to be updated during the training process. As the transformers operate on the token-level they produce token-level classifications. The MAE tool operates on the word level, so outputs on tokens were converted to outputs on words by applying the prediction from the first token of the word. Other methods, such as averaging the predictions or taking a maximum prediction across all tokens, gave similar results so the simpler method was favoured.

We experimented with different baseline pre-trained transformer models and found that BioBERT to be the strongest for both the tagging and linking tasks. We also considered having each predictor with its own transformer base but early experiments showed this to be less effective as there has little training data for each specific category. We tried this again with our final architecture and hyperparameters and found it to perform similarly. Having multiple predictors on one transformer also reduces the number of transformers needing to be fine-tuned, thus training far quicker.

In order to mitigate the effects of having a small number of positive examples in the data, loss balancing was employed in both models, where losses for infrequent tags were given a higher weighting; this includes the ‘None’ tag label.

We selected the entities derived from the group of 231 case study genes that were associated with both a perturbing action and at least one context, one effect, and one phenotype. Some genes were associated with multiple entities in the Context, Effect, and/or Phenotype groups. To calculate the precision of entity recognition, correctly recognized entities were manually scored as TP and entities that should not have been extracted were scored as FP. Perturbing actions, Genes, Contexts, Effects, and Phenotypes were scored as separate groups. Values for precision for all groups were >0.9 (Table 5). Correctly recognized entities were then scored for entity labelling, where entities labelled with the correct assertion(s) were scored as TP and entities labelled with one or more incorrect assertions were labelled as FP. These data were used to calculate scores for the precision of entity labelling, generating values of >0.9 for every group (Table 5). These results show that our model is capable of a high degree of precision in both the recognition and correct labelling of relevant biomedical entities (Table 5). Of note, in the Contexts group, the model achieved high precision values for both entity recognition (0.99) and labelling (0.93), despite the diverse and often idiosyncratic vocabulary used to describe different body tissues, cell lines, and animal strains.

We used the same case study dataset to examine the distribution of labelled assertions within the different groups. The Perturbing actions group contained representatives from all six assertion groups (as described in Table 1). ‘Gene loss-of-function’ occurred with the highest frequency (91/231) and ‘Pharmacological augmentation’ occurred with the lowest frequency (6/231).

The Contexts and Phenotypes groups both contained representatives from most, but not all, assertion groups.

In the Contexts group, the most frequently occurring assertion was ‘Cells’ (50/231) and the second most frequently occurring was ‘Cells; Organism’ (32/231), i.e. studies which involved at least two experimental contexts. The assertion ‘Transformed cells’ (i.e. cancer cells) was highly represented both alone (19/231) and in combination with other assertions, such as ‘Cells’ (5/231). These results show that we were able to extract entities corresponding to a broad range of experimental contexts. The assertion ‘Patient’ was not represented, either alone or in combination with other assertions. This likely reflects the fact that human gene perturbation experiments are more often carried out in either cell lines or patient-derived cells, which would be represented within either the ‘Cells’ or ‘Transformed cells’ assertion groups.

Within the Phenotype group, the most frequently represented assertion terms were ‘Apoptosis’ (32/231), ‘Proliferation’ (23/231), and ‘Tumourigenesis’ (20/231). Some assertion terms, including ‘Anoikis’, ‘Ferroptosis’, and ‘Quiescence’ were not represented (though a proportion of these were represented within the larger, unfiltered case study dataset of entities associated with 2919 genes). These results suggest that our model could be further refined by training it on a larger and more diverse corpus of annotated abstracts containing a broader range of phenotype terms.

In cancer genomics, it is of particular interest to identify those genes, which can induce cell death in cancer cells but which do not induce death of normal, healthy cells. To explore the potential of our methodology to identify such genes, we focused on the cell death phenotype ‘Apoptosis’, which was represented by a high number of entities within our case study data. Within the filtered dataset, there were 52 perturbing action and gene mentions associated with ’Apoptosis’ (Table 6). These genes were connected to a broad range of context assertion terms, consistent with the fact that apoptosis is a common form of cell death that has been observed in a many different experimental models.

The case study sample size was too small to allow in-depth analysis of gene perturbations in individual contexts. To further analyse the results, we therefore grouped the apoptosis-associated genes in two separate ways; (i) genes associated with in vitro assertion terms compared with genes associated with in vivo assertion terms and (ii) genes associated with non-cancer assertion terms and compared genes associated with cancer assertion terms (as detailed in Table 1). Within each of these two pairs of groups, there were some genes that were present within only one group and others which were present within both groups (Table 6). For example, in the in vitro group versus in vivo group pairing, the genes BCL-6, FasL, and BAX were associated with apoptosis in in vitro contexts only, Tctp, PKD1, and Ldlr were associated with apoptosis in in vivo contexts only, and the genes Ang2, Foxo3a, mTOR, Cox-2, and Smad2 were associated with apoptosis in both in vitro and in vivo contexts. In the non-cancer group versus cancer group pairing, the genes Ang2, CD28, and Foxo3a were associated with apoptosis in non-cancer contexts only, Pim1, Hsp90, and EGFR were associated with apoptosis in cancer contexts only, and p53, RAB25, and E2F-1 were associated with apoptosis in both non-cancer and cancer contexts. Thus, our methodology was able to identify genes involved in the regulation of apoptosis in a range of contexts and, when applied to a larger sample size, could be a useful means to explore variation of regulatory mechanisms in different cell types and experimental contexts.

Although the small sample size used in our case study precludes drawing any specific biological inferences, our results provide a proof-of-principle demonstration of how our new methodology could be used to generate insights into an important real-world question in cancer genomics.