RESEARCH ARTICLE Development of an Intervention Setting Ontology for behaviour change: Specifying where interventions take place [version 1; peer review: 2 approved] Emma Norris 1,2*, Marta M. Marques 1,3*, Ailbhe N. Finnerty1, Alison J. Wright 1, Robert West 4, Janna Hastings 1, Poppy Williams1, Rachel N. Carey1, Michael P. Kelly 5, Marie Johnston 6, Susan Michie 1 1Centre for Behaviour Change, University College London, London, UK 2Department of Clinical Sciences, Brunel University, Uxbridge, UK 3ADAPT SFI Research Centre, Trinity College Dublin, Dublin, Ireland 4Research Department of Epidemiology & Public Health, University College London, London, UK 5Primary Care Unit, Institute of Public Health, University of Cambridge, Cambridge, UK 6Aberdeen Health Psychology Group, University of Aberdeen, Aberdeen, UK * Equal contributors First published: 10 Jun 2020, 5:124 https://doi.org/10.12688/wellcomeopenres.15904.1 Latest published: 10 Jun 2020, 5:124 https://doi.org/10.12688/wellcomeopenres.15904.1 v1 Abstract Background: Contextual factors such as an intervention’s setting are key to understanding how interventions to change behaviour have their effects and patterns of generalisation across contexts. The intervention’s setting is not consistently reported in published reports of evaluations. Using ontologies to specify and classify intervention setting characteristics enables clear and reproducible reporting, thus aiding replication, implementation and evidence synthesis. This paper reports the development of a Setting Ontology for behaviour change interventions as part of a Behaviour Change Intervention Ontology, currently being developed in the Wellcome Trust funded Human Behaviour-Change Project. Methods: The Intervention Setting Ontology was developed following methods for ontology development used in the Human Behaviour- Change Project: 1) Defining the ontology’s scope, 2) Identifying key entities by reviewing existing classification systems (top-down) and 100 published behaviour change intervention reports (bottom-up), 3) Refining the preliminary ontology by literature annotation of 100 reports, 4) Stakeholder reviewing by 23 behavioural science and public health experts to refine the ontology, 5) Assessing inter-rater reliability of using the ontology by two annotators familiar with the ontology and two annotators unfamiliar with it, 6) Specifying ontological relationships between setting entities and 7) Making the Intervention Setting Ontology machine-readable using Web Ontology Language (OWL) and publishing online. Open Peer Review Reviewer Status Invited Reviewers 1 2 version 1 10 Jun 2020 report report Tracy Epton , University of Manchester, Manchester, UK 1. Chris Noone , National University of Ireland Galway, Galway, Ireland 2. Any reports and responses or comments on the article can be found at the end of the article.   Page 1 of 18 Wellcome Open Research 2020, 5:124 Last updated: 11 SEP 2020 Corresponding authors: Emma Norris (emma.norris@brunel.ac.uk), Marta M. Marques (MMOREIRA@tcd.ie), Susan Michie ( s.michie@ucl.ac.uk) Author roles: Norris E: Conceptualization, Data Curation, Formal Analysis, Investigation, Methodology, Project Administration, Resources, Software, Supervision, Validation, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing; Marques MM: Conceptualization, Data Curation, Formal Analysis, Investigation, Methodology, Project Administration, Resources, Software, Supervision, Validation, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing; Finnerty AN: Formal Analysis, Investigation, Methodology, Software, Visualization, Writing – Review & Editing; Wright AJ: Investigation, Methodology, Writing – Review & Editing; West R: Conceptualization, Funding Acquisition, Investigation, Methodology, Writing – Review & Editing; Hastings J: Methodology, Software, Writing – Review & Editing; Williams P: Investigation, Writing – Review & Editing; Carey RN: Conceptualization, Investigation, Writing – Review & Editing; Kelly MP: Conceptualization, Funding Acquisition, Investigation, Writing – Review & Editing; Johnston M: Conceptualization, Investigation, Methodology, Writing – Review & Editing; Michie S: Conceptualization, Funding Acquisition, Methodology, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing Competing interests: No competing interests were disclosed. Grant information: This work is supported by the Wellcome Trust through a collaborative award to the Human Behaviour-Change Project [201524]. MMM is funded by a Marie-Sklodowska-Curie fellowship [EU H2020 EDGE program grant agreement No. 713567]. Copyright: © 2020 Norris E et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. How to cite this article: Norris E, Marques MM, Finnerty AN et al. Development of an Intervention Setting Ontology for behaviour change: Specifying where interventions take place [version 1; peer review: 2 approved] Wellcome Open Research 2020, 5:124 https://doi.org/10.12688/wellcomeopenres.15904.1 First published: 10 Jun 2020, 5:124 https://doi.org/10.12688/wellcomeopenres.15904.1 Results: The Intervention Setting Ontology consists of 72 entities structured hierarchically with two upper-level classes: Physical setting including Geographic location, Attribute of location (including Area social and economic condition, Population and resource density sub-levels) and Intervention site (including Facility, Transportation and Outdoor environment sub-levels), as well as Social setting. Inter-rater reliability was found to be 0.73 (good) for those familiar with the ontology and 0.61 (acceptable) for those unfamiliar with it. Conclusion: The Intervention Setting Ontology can be used to code information from diverse sources, annotate the setting characteristics of existing intervention evaluation reports and guide future reporting. Keywords ontology, behaviour change, context, evidence synthesis, intervention reporting, stakeholder review This article is included in the Human Behaviour- Change Project collection.   Page 2 of 18 Wellcome Open Research 2020, 5:124 Last updated: 11 SEP 2020 Introduction Effects of interventions to improve health vary considerably across contexts of settings and target populations. While this is widely acknowledged in the literature, the specific elements in the context and their mechanisms of action on outcomes are either assumed or obscure (Michie et al., 2017). In order to under- stand this variation arising from the different aspects of context, it is helpful to synthesise evidence about the ways in which these modifying variables influence intervention effectiveness. This requires detailed and consistent specification of study contexts. There are many different classification systems and ontologies describing interventions, including their settings and target populations; however, these have limitations such as incomplete coverage and relevance across the range of international contexts. In this paper, we consider intervention setting. A forthcoming paper will report the development of an Intervention Population Ontology (Finnerty et al., In preparation). Intervention settings are not currently consistently reported with enough specificity or comprehensiveness to allow accurate replication. The CONsolidated Standards of Reporting Trials statement (CONSORT; Schulz et al., 2010) includes one item referring to setting (Item 4b – Settings and locations where the data were collected), with its extension for social and psychology interventions CONSORT-SPI (Montgomery et al., 2018) adding an additional item (Item 4b – Where applicable, eligibility criteria for settings and those delivering the intervention). The Template for Intervention Description and Replication checklist (TIDieR; Hoffman et al., 2014) includes one item for setting (Item 7 – Where: describe the type(s) of location(s) where the intervention occurred, including any necessary infrastructure or relevant features). The recent Typology of Interventions in Proximal Physical Micro-Environments typology (TIPPME: Hollands et al., 2017) allows specification of micro-level aspects of the physical environment related to behaviours. Although this was based on an exhaustive review of the literature, TIPPME is restricted to interventions in micro- environments or contexts aimed at changing selection, purchase and consumption of food, alcohol and tobacco. We currently lack a classification system to aid researchers in describing in detail, and using shared language, the variety of settings of behaviour change interventions (BCIs) or indeed behaviour more broadly. What at first sight would seem to be a fairly straightforward task of describing intervention settings is actually very complex, given the diversity of entities, terms and definitions across academic disciplines, employment sectors and cultures. Ontologies are a tool for addressing this diversity by enabling ‘semantic inter-operability’ by associating computational data with unam- biguous shared meaning (Hastings, 2017; Michal et al., 2012). Ontologies are data structures that enable precise specification of knowledge in a given domain (Arp et al., 2015). In infor- mation science, ontologies provide a set of: i) unique and unambiguous identifiers representing types of entity (such as objects, attributes or processes), ii) labels and definitions corresponding to these identifiers, and iii) specified relationships between the entities (Arp et al., 2015; Larsen et al., 2017; Norris et al., 2019). These labels, definitions and relationships comprise a ‘controlled vocabulary’ and formal specification for the given domain. Ontologies are dynamic representations that are maintained and updated according to new evidence about entities and relationships (He et al., 2018). Machine-readable ontologies provide an excellent structure for annotating scientific reports to allow evidence synthesis (Michie & Johnston, 2017). As seen in other fields such as genetics (Ashburner et al., 2000), the availability and use of ontologies allows an active, iteratively developed basis for shared knowledge and understanding (Michie & Johnston, 2017). As machine- readable artefacts, ontologies can be harnessed for annotation and evidence synthesis, such as the automation of literature searching, statistical analysis workflows and database searching and browsing, as well as in other computational applications (Hastings, 2017) (see glossary of italicised terms in Table 1). As yet, no ontology exists to describe the complexity of behaviour change intervention settings (Norris et al., 2019). A comprehensive Behaviour Change Intervention Ontology (BCIO) is being developed as part of the Human Behaviour-Change Project (Michie et al., 2017). The BCIO consists of an upper level with 42 entities, one of which is Behaviour change intervention setting, specified as part of the Context in a given BCI scenario (Michie et al., 2020). Drawing on the methodology used to develop a taxonomy of behaviour change techniques (BCTTv1; Michie et al., 2017) and other relevant ontologies (Norris et al., 2019), the current study aimed to develop an ontology for specifying and classifying characteristics of the settings in which interventions take place. These settings are generally applicable beyond the scope of behaviour change interventions. This paper reports the development and final version of the Intervention Setting Ontology. Methods The Intervention Setting Ontology was developed in an iterative process of seven steps (Wright et al., 2020). Step 1 – Defining the scope of the Intervention Setting Ontology A definition and overall topic for the ontology was set by reviewing dictionaries and the reporting guidelines of CONSORT (Schulz et al., 2010), CONSORT-SPI (Montgomery et al., 2018), TIDieR (Hoffmann et al., 2014) and TIPPME (Hollands et al., 2017). Step 2 – Identifying key entities and developing the preliminary Intervention Setting Ontology An initial prototype version of the ontology was developed using both a bottom-up and top-down approach. In the bottom-up approach, 100 published reports of BCIs were reviewed to develop an initial list of intervention setting charac- teristics. These reports were randomly selected from a larger dataset of BCI reports partially annotated for behaviour change techniques, mechanisms of action, and modes of delivery, covering a range of health behaviours (Carey et al., 2019; Michie et al., 2015). In the top-down approach, existing classification systems of intervention setting characteristics were identified from: i) pub- lished ontologies containing terms related to behaviour change Page 3 of 18 Wellcome Open Research 2020, 5:124 Last updated: 11 SEP 2020 Ta bl e 1. G lo ss ar y. Te rm D efi ni tio n So ur ce A nn ot at io n Pr oc es s of c od in g se le ct ed p ar ts o f d oc um en ts o r o th er re so ur ce s to id en tif y th e pr es en ce o f o nt ol og y en tit ie s M ic hi e et a l., 2 01 7. A nn ot at io n gu id an ce m an u al W rit te n gu id an ce o n ho w to id en tif y an d ta g pi ec es o f t ex t f ro m in te rv en tio n ev al ua tio n re po rt s w ith s pe ci fic c od es re la tin g to e nt iti es in th e on to lo gy , u si ng EP PI -R ev ie w er s of tw ar e. B as ic F or m al O nt ol og y (B FO ) A n up pe r l ev el o nt ol og y co ns is tin g of c on tin ua nt s an d oc cu rr en ts d ev el op ed to su pp or t i nt eg ra tio n, e sp ec ia lly o f d at a ob ta in ed th ro ug h sc ie nt ifi c re se ar ch . A rp e t a l., 2 01 5. En tit y A ny th in g th at e xi st s, th at c an b e a co nt in ua nt o r a n oc cu rr en t a s de fin ed in th e B as ic F or m al O nt ol og y. A rp e t a l., 2 01 5. EP PI -R ev ie w er A w eb -b as ed s of tw ar e pr og ra m fo r m an ag in g an d an al ys in g da ta in a ll ty pe s of sy st em at ic re vi ew (m et a- an al ys is , f ra m ew or k sy nt he si s, th em at ic s yn th es is e tc . I t m an ag es re fe re nc es , s to re s PD F fil es a nd fa ci lit at es q ua lit at iv e an d qu an tit at iv e an al ys es s uc h as m et a- an al ys is a nd th em at ic s yn th es is . I t a ls o ha s a fa ci lit at e to an no ta te p ub lis he d pa pe rs . Th om as & B ru nt on , 2 01 0. EP PI -R ev ie w er 4 : h ttp :// ep pi .io e. ac .u k/ ep pi re vi ew er 4/ EP PI -R ev ie w er W eb V er si on : h ttp s: //e pp i.i oe .a c. uk /e pp ire vi ew er -w eb / G itH ub A w eb -b as ed p la tfo rm u se d as a re po si to ry fo r s ha rin g co de , a llo w in g ve rs io n co nt ro l. ht tp s: //g ith ub .c om / In te r-r at er re lia bi lit y St at is tic al a ss es sm en t o f s im ila rit y an d di ss im ila rit y of c od in g be tw ee n tw o or m or e co de rs . I f i nt er -r at er re lia bi lit y is h ig h th is s ug ge st s th at o nt ol og y en tit y de fin iti on s an d la be ls a re b ei ng in te rp re te d si m ila rly b y th e co de rs . G w et , 2 01 4. H an db oo k of in te r-r at er re lia bi lit y: T he d efi ni tiv e gu id e to m ea su rin g th e ex te nt o f a gr ee m en t a m on g ra te rs . G ai th er sb ur g, A dv an ce d A na ly tic s. In te ro pe ra bi lit y O nt ol og y de ve lo pe rs s ho ul d co lla bo ra te w ith o th er s w he re ve r p os si bl e to re -u se en tit ie s an d lim it du pl ic at io n of w or k. In te ro pe ra bi lit y of o nt ol og ie s si ts w ith in th e O B O F ou nd ry p rin ci pl e of C om m itm en t t o C ol la bo ra tio n. ht tp :// w w w .o bo fo un dr y. or g/ pr in ci pl es /fp -0 10 -c ol la bo ra tio n. ht m l Is su e tr ac ke r A n on lin e lo g fo r p ro bl em s id en tifi ed b y us er s ac ce ss in g an d us in g an o nt ol og y. B C IO Is su e Tr ac ke r: ht tp s: //g ith ub .c om / H um an B eh av io ur C ha ng eP ro je ct /o nt ol og ie s/ is su es O BO F ou nd ry Th e O pe n B io lo gi ca l a nd B io m ed ic al O nt ol og y (O B O ) F ou nd ry is a c ol le ct iv e of o nt ol og y de ve lo pe rs th at a re c om m itt ed to c ol la bo ra tio n an d ad he re nc e to sh ar ed p rin ci pl es . T he m is si on o f t he O B O F ou nd ry is to d ev el op a fa m ily o f in te ro pe ra bl e on to lo gi es th at a re b ot h lo gi ca lly w el l-f or m ed a nd s ci en tifi ca lly ac cu ra te . Sm ith e t a l., 2 00 7; w w w .o bo fo un dr y. or g/ O nt ol og y A s ta nd ar di se d re pr es en ta tio na l f ra m ew or k pr ov id in g a se t o f t er m s fo r t he co ns is te nt d es cr ip tio n (o r “ an no ta tio n” o r “ ta gg in g” ) o f d at a an d in fo rm at io n ac ro ss d is ci pl in ar y an d re se ar ch c om m un ity b ou nd ar ie s. A rp e t a l., 2 01 5. Pa re n t c la ss A s ub su m in g cl as s w ith in a n on to lo gy th at is re la te d to o ne o r m or e ch ild (s ub su m ed ) c la ss es . A rp e t a l., 2 01 5. RO BO T A n au to m at ed c om m an d lin e to ol fo r o nt ol og y w or kfl ow s. Ja ck so n et a l., 2 01 9; h ttp :// ro bo t.o bo lib ra ry .o rg UR I A s tri ng o f c ha ra ct er s th at u na m bi gu ou sl y id en tifi es a n on to lo gy o r a n in di vi du al en tit y w ith in a n on to lo gy . H av in g U R I i de nt ifi er s is o ne o f t he O B O F ou nd ry pr in ci pl es . ht tp :// w w w .o bo fo un dr y. or g/ pr in ci pl es /fp -0 03 -u ris .h tm l W eb O nt ol og y La ng ua ge (O W L) A fo rm al la ng ua ge fo r d es cr ib in g on to lo gi es . I t p ro vi de s m et ho ds to m od el cl as se s of “ th in gs ”, h ow th ey re la te to e ac h ot he r a nd th e pr op er tie s th ey h av e. O W L is d es ig ne d to b e in te rp re te d by c om pu te r p ro gr am s an d is e xt en si ve ly us ed in th e Se m an tic W eb w he re ri ch k no w le dg e ab ou t w eb d oc um en ts a nd th e re la tio ns hi ps b et w ee n th em a re re pr es en te d us in g O W L sy nt ax . ht tp s: //w w w .w 3. or g/ TR /o w l2 -q ui ck -r ef er en ce / Page 4 of 18 Wellcome Open Research 2020, 5:124 Last updated: 11 SEP 2020 intervention setting via the Ontology Lookup Service and BioPortal; ii) the Patient, Intervention, Comparison, Outcome (‘PICO’) ontology developed by the Cochrane Collaboration due to its relevance for intervention trials; and iii) controlled medical vocabularies (e.g. SNOMED CT, MedDRA, MeSH). The preliminary ontology contained a label and definition for each entity representing an intervention setting characteristic. Definitions were developed using pre-specified guidance, with the standard format of definitions being: A is a B that C, or involves or relates to C in some way, where A is the class being defined, B is a parent class and C describes a set of properties of A that distinguish it from other members of B (Michie et al., 2019). It was piloted with published BCI reports focusing on smoking cessation and physical activity behaviours (Michie et al., 2017). BCI reports were annotated independently by two researchers in batches of 10, with each entity annotated as either present or absent. Two types of inter-rater reliability measures were used: i. percentage of agreement between cod- ers and ii. Cohen’s Kappa (Cohen, 1960). Kappa statistics are only reported in instances where the researchers allocated a code to at least five cases (Michie et al., 2015). Satisfactory inter- rater reliability was achieved by the time 55 papers had been coded. After this, no additional adjustments were made to the prototype version of the Intervention Setting Ontology. Step 3 – Refinement of the ontology through literature annotation, discussion and revision The preliminary ontology was revised by the research team based on the results of the pilot annotations. Using EPPI-Reviewer 4 software (Thomas & Brunton, 2010), two researchers independ- ently annotated 30 BCI reports on smoking cessation interven- tions using the revised Intervention Setting Ontology. An open alternative to this software used for annotation is PDFAnno (Shindo et al., 2018). Discrepancies were discussed and the ontology structure, definitions and annotation guidance man- ual were revised. A second set of annotators followed the same procedure for another set of 45 BCI reports of smoking cessa- tion, and 40 BCI reports of physical activity. All reports were randomised controlled trials from one of three datasets: Cochrane Reviews, papers annotated for behaviour change techniques and papers from the IC-SMOKE project (Black et al., 2020; De Bruin et al., 2016) (List of papers used in development of ontology: https://osf.io/4qcby/ (West et al., 2020)). Step 4 – Expert stakeholder review Ninety-eight members of a panel of behavioural scientists and public health expert stakeholders were invited to give feedback on the Intervention Setting Ontology resulting from Step 3. These experts comprised i) 65 behavioural scientists who had provided feedback on previous projects at the Centre for Behav- iour Change, ii) 16 experts from under-represented countries identified through the BCTTv1 database, and iii) 17 stakeholders who expressed interest in being involved in the Human Behaviour-Change Project stakeholder initiatives. Experts from both ‘well-represented’ countries (UK, USA, Canada, Australia, the Netherlands) and other ‘less-represented’ countries were randomly selected to provide feedback using Researcher Randomizer. Feedback was collected through an online questionnaire, using QualtricsTM software (Full survey https://osf.io/8audy/ (West et al., 2020)), with the task designed to take no longer than 45 minutes to complete. The task asked experts to: 1. identify the characteristics of intervention setting that were of interest to them when trying to understand variation in the effectiveness of BCIs (open-ended question). Experts were advised to consider a specific behaviour when answering this question e.g ‘physical activity’ 2. rate the importance of each of the setting entities on a 5-point Likert scale (1 = “not important”, 2 = “slightly important”, 3 = “moderately important”, 4 = “important”, 5 = “very important” or “don’t know/not sure”). For example: “How important do you think each of the following Geographic location characteristics are to understand variation in the effectiveness of at least some behaviour change interventions?” (Country of intervention & Within country location), and 3. provide feedback on the completeness and comprehensiveness of the Setting Ontology. Experts were also asked to indicate: i) if there were any enti- ties missing (If yes, which should be added), ii) if there were any entities or definitions that should be changed (if yes, what changes should be considered), and iii) If there were any entities that should be placed in a different location in the classification hierarchy of the Intervention Setting Ontology. A thematic analysis of the responses was conducted and means and standard deviations of ratings were calculated. The feed- back from the expert consultation was discussed by the research team and the Intervention Setting Ontology and annotation guidance were revised. Step 5 – Inter-rater Reliability of Annotations using the Intervention Setting Ontology Assessment of inter-rater reliability of the annotations by two researchers leading the development of the ontology was con- ducted using 50 papers from Cochrane reviews (30 for smoking cessation and 20 for physical activity). Inter-rater reliability was also assessed for annotations by two behaviour change experts unfamiliar with the ontology but with experience in annotat- ing BCI reports. Annotation was of a random sample of 50 randomised controlled trials from a database of papers coded by Behaviour Change Techniques, with no restrictions on the out- come behaviour. Inter-rater reliability was assessed using Krip- pendorff’s Alpha (Hayes & Krippendorff, 2007) using Python 3.6 (https://github.com/HumanBehaviourChangeProject/Automa- tion-InterRater-Reliability) (Finnerty & Moore, 2020), as unlike Cohen’s Kappa, Krippendorff factors in both agreement and disagreement within annotations. Step 6 – Specifying the relationships between Intervention Setting Ontology entities The research team established relationships between ontology entities to formalise the knowledge present in the ontology. This process was conducted in line with Basic Formal Ontology Page 5 of 18 Wellcome Open Research 2020, 5:124 Last updated: 11 SEP 2020 principles which have been used extensively in biomedical ontologies (Arp et al., 2015). The suitability of common rela- tionships from Basic Formal Ontology (Arp et al., 2015) and the Relation Ontology (Smith et al., 2005) were assessed, includ- ing the basic hierarchical relationship ‘is_a’ which holds between classes where one class is a subclass of another class, and ‘located_in’, which relates an entity to a spatial region demarcating a location. Step 7 - Making the Intervention Setting Ontology machine-readable and available online The Intervention Setting Ontology was initially developed as a table of entities, with separate rows for each entity annotated with a primary label, definition, synonyms and relationships. When the Intervention Setting Ontology was at a stable level of development for initial release, it was converted into the Web Ontology Language (OWL) (Antoniou & Van Harmelen, 2004) format, enabling it to be viewed and visualised using ontology software such as Protégé and to be compatible with other ontolo- gies. The conversion to OWL used the ROBOT ontology toolkit library (Jackson et al., 2019), which provides a facility to create well-formatted ontologies from templates. A ROBOT template is a comma-separated values (CSV) file that can be prepared eas- ily in common spreadsheet software, annotated with instruc- tions for translation from spreadsheet columns to OWL language and metadata attributes. Within the input template spreadsheet, separate columns represent the entity ID (e.g. BCIO_0013), name, definition, relationship with other entities, examples and synonyms. This OWL version of the Intervention Setting Ontology was then stored on the project GitHub repository, as GitHub has an issue tracker which allows feedback to be submitted by mem- bers of the community which can be responded to, and if neces- sary, addressed in subsequent releases. When the full Behaviour Change Intervention Ontology has been confirmed, it will be submitted to the OBO Foundry (Smith et al., 2007). Results Step 1 – Defining the scope of the Intervention Setting Ontology Given that ‘setting’ is defined in a general lexicon as ‘the place or type of surroundings where something is positioned or where an event takes place’, an intervention’s setting was defined more precisely as ‘An aggregate of entities that form the environment in which a BCI is provided.’ Step 2 - Identifying key entities and developing the preliminary Intervention Setting Ontology The initial prototype version of the Intervention Setting Ontol- ogy encompassed a four-level hierarchical structure, contain- ing 76 unique entities (https://osf.io/g8qfv/ (West et al., 2020)). Inter-rater agreement for identifying the presence of a setting entity was low in terms of percentage, at 45.5%. Kappa statistics varied from ‘perfect’ for entities such as Accommodation to low agreement (κ=0.300) for entities such as Community setting. Step 3 – Refinement of the Intervention Setting Ontology Based on the annotations from Step 2, changes were made to the ontology. Two terms, ‘particular‘ and ’unclear/not reported‘, were deleted as they did not meet the ontological require- ment of being unique discrete entities with corresponding definitions and attributes (Arp et al., 2015). Other changes were: 1) Health Care facility was revised from having the lower-level entities Primary care, Secondary Care, Tertiary Care, Phar- macy and Hospice, to having lower-level entities of Hospital facility, Doctor-led primary care facility, Care home facil- ity, Hospice facility, Psychiatric facility, Pharmacy facility, Community health care facility and Dentist facility; 2) Public transportation was extended from only public trans- portation to a new entity named Transportation which includes Public transportation, Mobile intervention venue as well as Private transportation; 3) Outdoor environment was added to the ontology; 4) Attribute of location was added to the ontology, including new entities Area social and economic condition and Popula- tion and resource distribution (previously placed in Geographic location). Changes to labels and definitions were made to reflect the structural changes. Step 4 – Expert stakeholder review Of the 98 experts contacted, 78 were from ‘well-represented’ coun- tries and 20 from ‘less-represented’ countries. Of the 23 experts (23.5%) completing the survey, 19 were from ‘well-represented’ and four from ‘less-represented’ countries. Experts’ responses and how these were addressed within the ontology development are reported at: https://osf.io/npsy7/ (West et al., 2020). The setting entities rated as of top importance by experts were Area social and economic condition (M=4.28/5; SD=0.87), Outdoor environment (M=4.28; SD=1.24), Healthcare facility (M=4.22; SD=0.79), Educational facility (M=4.06; SD=0.85), Transportation (M=4.06; SD=1.27) and Community facility (M=-4.00; SD=1.00). Changes made to the Intervention Setting Ontology as a result of stakeholder feedback included adding Suburban area den- sity, Developed- and Developing country and expanding exam- ples within Sport and exercise facility such as swimming pool and stadium. Suggestions to add eHealth or mHealth intervention descriptors (n=3) were not incorporated in the Intervention Setting Ontology, as these are classified in the Modes of Delivery ontol- ogy (Marques et al., 2020) within the wider Behaviour Change Intervention Ontology (https://osf.io/h4sdy/ (West et al., 2020)). Some suggested changes were not made as they would have decreased the generalisability of the Intervention Set- ting Ontology. For example, a suggestion to add a variety of school types such as Voluntary Aided (VA), State, Private, Faith, Academies etc would have led to UK-specific terminol- ogy (UK Government, 2019). The broad approach of classifying Page 6 of 18 Wellcome Open Research 2020, 5:124 Last updated: 11 SEP 2020 school settings as Primary, Middle or Secondary school was maintained to capture the range of international school settings. Step 5 – Inter-rater reliability of annotations using the Intervention Setting Ontology Inter-rater reliability from the 50 papers annotated by those familiar with the ontology was found to be good (a=0.73). The random selection of 50 papers used for inter-rater reliability test- ing in those unfamiliar with the ontology resulted in papers with the following target behaviours: physical activity (k=16), dietary behaviours (k=9), sexual behaviours (k=8), alcohol (k=7) and other behaviours such as medication adherence (k=11). The inter-reliability for these annotations was acceptable (a=0.61) (Hayes & Krippendorff, 2007). Step 6 – Specifying the relationships between Intervention Setting Ontology entities Relationships from the Relation Ontology (Smith et al., 2005) were used to connect classes, namely the basic hierarchi- cal relationship ‘is_a’ which holds between classes where one class is a subclass of another class, and ‘is_attribute_of’ which holds between classes where one class is a quality or feature of the other. Step 7 – Making the Intervention Setting Ontology machine-readable and available online A downloadable version of the final Intervention Setting Ontol- ogy is available from GitHub (Norris et al., 2020). The hierar- chical structure, URIs, labels and definitions for all entities are described in Table 2. The ontology is accompanied by an annota- tion guidance manual that provides guidance on how to annotate for these entities in BCI reports (available at https://osf.io/76jty/) (West et al., 2020). The final version of the Intervention Setting Ontology presents a six-level hierarchical structure comprising of 72 unique enti- ties. There are two upper-level classes: Physical setting (BCIO: 026000: A physical environment in which a BCI is delivered) and Social setting (BCIO: 029000: An aggregate of people with whom a BCI population interacts). Physical setting includes Geographic location (GAZ:00000448: A reference to a place on the Earth, by its name or by its geographic location, used from the existing Gazetteer Ontology), Attribute of location (BCIO: 026003: Features of a given location, such as social and economic characteristics) and Site (BFO_0000029: A three- dimensional immaterial entity that is (partially or wholly) bounded by a material entity or it is a three-dimensional immaterial part thereof). For each of these entities, there are lower-level entities that inherit its properties. For example Site includes: Facility (OMRSE:00000062, used from the existing Ontology of Medi- cally Related Social Entities; Hicks et al., 2016), Transportation (NCIT_C141286, from the NCI Thesaurus OBO Edition; Balhoff et al., 2017) and Outdoor environment (BCIO:026044). Facility includes subclasses of Residential facility (OMRSE:00000191), Healthcare facility (OMRSE:00000102), Educational facility (BCIO:026022), Community facility (BCIO:026029), Retail facility (BCIO: 026036), Research facil- ity (ENVO:00000469 from the Environment Ontology; Buttigieg et al., 2013), Office facility (BCIO:026037), Criminal justice facility (BCIO:026038), Factory facility (BCIO:026039), and Military facility (BCIO:026040). Residential facility within Facil- ity includes subclasses of Household residence (BCIO:026009), Multiple occupancy residence (BCIO:026010), Homeless setting (BCIO:026013) and Temporary residence (BCIO: 026014). Finally, at the lowest level, Multiple occupancy residence within Residen- tial facility has subclasses of Student residence (BCIO:026011) and Residential care or assisted living (BCIO:026012). Discussion This study developed the Intervention Setting Ontology to spec- ify formally the characteristics of the settings in which behav- iour change interventions (BCIs) take place, as part of the Behaviour Change Intervention Ontology (Michie et al., 2017). Although developed primarily to specify settings of behaviour change interventions, the settings are generally applicable to other types of intervention or contexts. The ontology consists of 72 entities structured hierarchically with two upper-level classes: Physical setting (BCIO:026000: A physical environment in which a BCI is delivered) and Social setting (BCIO:029000: An aggregate of people with whom a BCI population inter- acts). Physical setting is further sub-divided by three upper-level classes: Geographic location, Attribute of location (includ- ing Area social and economic, Population and resource density sub-levels) and Site (including Facility, Transportation and Outdoor environment sub-levels). Inter-rater reliability was found to be 0.73 (good) for those familiar with the ontology and 0.61 (acceptable) for those unfamiliar with it, as assessed by Krippendorff’s alpha. Together with ‘population’, it makes up Context which is part of a wider set of lower-level ontolo- gies within the Behaviour Change Intervention Ontology (BCIO). The ontologies within the BCIO are connected to each other by specified relationships. For example, the contextual entity of Intervention Setting is related to the contextual entity of Population: who receives an intervention (Finnerty et al., In preparation). In addition, entities within the Intervention Setting Ontology can be integrated or linked to ontologies beyond the BCIO, a key feature of OWL ontologies which encourages re-use and adoption (Hastings, 2017). Ontologies should be dynamic representations that are main- tained and updated according to new evidence about entities and relationships (Arp et al., 2015; He et al., 2018). The Inter- vention Setting Ontology and all other ontologies within the Human Behaviour-Change Project will be updated as they are informed by advances in behavioural science and by online feedback from ontology users via the GitHub portal. Strengths and limitations Domain experts are often not formally consulted when ontolo- gies are developed (Norris et al., 2019), with the result that development may be restricted to the knowledge, thinking Page 7 of 18 Wellcome Open Research 2020, 5:124 Last updated: 11 SEP 2020 Ta bl e 2. En tit y la be ls , d efi ni tio ns a nd U RI s fo r al l I nt er v en tio n Se tti ng O nt ol og y En tit ie s. Up pe r-L ev el Su b- Le v el 1 Su b- Le v el 2 Su b- Le v el 3 Su b- Le v el 4 Su b- Le v el 5 D efi ni tio n Ph ys ic al s et tin g B C IO :0 26 00 0 A p hy si ca l e nv iro nm en t i n w hi ch a B C I i s de liv er ed . G eo gr ap hi c lo ca tio n G A Z_ 00 00 04 48 A re fe re nc e to a p la ce o n th e Ea rt h, b y its na m e or b y its g eo gr ap hi ca l l oc at io n C ou nt ry o f i nt er ve nt io n B C IO :0 26 00 1 A g eo gr ap hi c lo ca tio n of a c ou nt ry w he re th e in te rv en tio n ta ke s pl ac e. W ith in -c ou nt ry lo ca tio n B C IO :0 26 00 2 A g eo gr ap hi c lo ca tio n w ith in a c ou nt ry w he re th e in te rv en tio n ta ke s pl ac e. Ex am pl e: re gi on , t ow n, c ity , s ta te A ttr ib ut e of lo ca tio n B C IO :0 26 00 3 Fe at ur es o f a g iv en lo ca tio n, s uc h as s oc ia l an d ec on om ic c ha ra ct er is tic s. A re a so ci al a nd ec on om ic c on di tio n B C IO :0 26 00 4 A n at tri bu te o f l oc at io n de sc rib in g th e ov er al l so ci oe co no m ic s ta te o f a lo ca tio n. Ex am pl e: d is ad va nt ag ed a re a, c rim e ra te s, W or ld B an k C la ss ifi ca tio ns Lo w -in co m e ar ea B C IO :0 26 00 5 A n ar ea s oc ia l a nd e co no m ic c on di tio n de sc rib ed to h av e a lo w -in co m e, w he th er a t a co un tr y or w ith in -c ou nt ry le ve l. Ex am pl e: d ev el op in g co un tr y H ig h- in co m e ar ea B C IO :0 26 00 6 A n ar ea s oc ia l a nd e co no m ic c on di tio n de sc rib ed to h av e a hi gh - i nc om e, w he th er a t a co un tr y or w ith in -c ou nt ry le ve l. Ex am pl e: d ev el op ed c ou nt ry Po pu la tio n an d re so ur ce d en si ty B C IO :0 02 60 07 A n at tri bu te o f l oc at io n de sc rib in g th e de ns ity of a n ar ea , i n te rm s of p eo pl e an d re so ur ce s w ith in it . R ur al a re a EN VO :0 10 00 77 2 A n ar ea w hi ch is o ut si de o f a to w n, c ity , o r ur ba n ar ea . R ur al a re as a re p rim ar ily u se d fo r ag ric ul tu re o r p as to ra lis m a nd m ay c on ta in ru ra l s et tle m en ts Su bu rb an a re a B C IO : 0 26 00 8 A n ar ea o n th e ed ge o f a la rg e to w n or c ity w he re a p ro po rt io n of th os e w ho w or k in th e to w n or c ity li ve U rb an a re a EN VO :0 10 00 85 6 In co rp or at ed p op ul at ed p la ce Si te B FO _0 00 00 29 A th re e- di m en si on al im m at er ia l e nt ity th at is (p ar tia lly o r w ho lly ) b ou nd ed b y a m at er ia l en tit y or it is a th re e- di m en si on al im m at er ia l pa rt th er eo f Fa ci lit y O M R SE :0 00 00 06 2 A n ar ch ite ct ur al s tru ct ur e th at b ea rs s om e fu nc tio n. Page 8 of 18 Wellcome Open Research 2020, 5:124 Last updated: 11 SEP 2020 Up pe r-L ev el Su b- Le v el 1 Su b- Le v el 2 Su b- Le v el 3 Su b- Le v el 4 Su b- Le v el 5 D efi ni tio n R es id en tia l f ac ili ty O M R SE _0 00 00 19 1 A fa ci lit y th at h as a t l ea st o ne h ou si ng u ni t a s pa rt in w hi ch a p er so n or p er so ns li ve H ou se ho ld re si de nc e B C IO :0 26 00 9 A fa ci lit y w he re a n in di vi du al is li vi ng a lo ne o r w ith o ne o r m or e pe rs on . T he in di vi du al s do no t h av e to b e re la te d M ul tip le o cc up an cy re si de nc e B C IO :0 26 01 0 A fa ci lit y w he re a n in di vi du al li ve s w ith m an y ot he rs th at m ay b e co lle ct ed a cc or di ng to a so ci al s tru ct ur e su ch a s ed uc at io n or d is ab ili ty . Ex am pl e: b ed si t St ud en t r es id en ce B C IO :0 26 01 1 A m ul tip le o cc up an cy re si de nc e w he re m an y st ud en ts li ve . Ex am pl e: s tu de nt h al ls R es id en tia l c ar e or as si st ed li vi ng B C IO :0 26 01 2 A m ul tip le o cc up an cy re si de nc e w he re m ul tip le v ul ne ra bl e pe op le li ve . Ex am pl e: re tir em en t h om e H om el es s se tti ng B C IO :0 26 01 3 A re si de nt ia l f ac ili ty w he re a n in di vi du al is liv in g th at is n ot s ta bl e an d se cu re . Ex am pl e: a n ar ch ite ct ur al s tru ct ur e fo r w hi ch an in di vi du al d oe s no t h av e a le ga l r ig ht to st ay Te m po ra ry re si de nc e B C IO :0 26 01 4 A re si de nt ia l f ac ili ty w he re in di vi du al s ar e in a tra ns iti on al s ta te o f h ou si ng a nd n ot s ta yi ng fo r a pr ol on ge d pe rio d. Ex am pl e: H os te ls , B & B , e m er ge nc y ac co m m od at io n, H ea lth ca re fa ci lit y O M R SE :0 00 00 10 2 A fa ci lit y th at is a dm in is te re d by a h ea lth c ar e or ga ni sa tio n fo r t he p ur po se o f p ro vi di ng he al th c ar e to a p at ie nt p op ul at io n H os pi ta l f ac ili ty O M R SE :0 00 00 06 3 A fa ci lit y th at is ru n by a h os pi ta l o rg an iz at io n, su ch a s em er ge nc y de pa rt m en ts , o ut pa tie nt cl in ic s an d re ha bi lit at io n an d is th e be ar er o f a ho sp ita l f un ct io n Em er ge nc y de pa rt m en t f ac ili ty O M R SE :0 00 00 11 4 A h ea lth c ar e fa ci lit y th at b ea rs a fu nc tio n to pr ov id e em er ge nc y he al th ca re s er vi ce s an d th e ac ut e ca re o f p at ie nt s w ho p re se nt w ith ou t pr io r a pp oi nt m en t, ha vi ng a rr iv ed e ith er b y th ei r o w n m ea ns o r b y am bu la nc e H os pi ta l o ut pa tie nt cl in ic fa ci lit y B C IO :0 26 01 5 A h os pi ta l f ac ili ty to tr ea t p at ie nt s w ith ou t t he m st ay in g ov er ni gh t, of te n af te r a h os pi ta l v is it D oc to r-l ed p rim ar y ca re fa ci lit y B C IO :0 26 01 6 A h ea lth ca re fa ci lit y le d by d oc to rs . Ex am pl e: g en er al p ra ct iti on er s ur ge ry , d oc to rs ’ su rg er y Page 9 of 18 Wellcome Open Research 2020, 5:124 Last updated: 11 SEP 2020 Up pe r-L ev el Su b- Le v el 1 Su b- Le v el 2 Su b- Le v el 3 Su b- Le v el 4 Su b- Le v el 5 D efi ni tio n C ar e ho m e fa ci lit y B C IO :0 26 01 7 A h ea lth ca re fa ci lit y th at is ru n by a c ar e ho m e or ga ni za tio n an d is th e be ar er o f a c ar e ho m e fu nc tio n. H os pi ce fa ci lit y O M R SE :0 00 00 10 4 A h ea lth c ar e fa ci lit y th at b ea rs a fu nc tio n to pr ov id e he al th ca re to th e si ck o r t er m in al ly il l Ph ar m ac y fa ci lit y PD R O :0 00 00 74 A h ea lth c ar e fa ci lit y w ho se fu nc tio n is to s to re , pr ep ar e, d is pe ns e, a nd m on ito r t he u sa ge o f ph ar m ac eu tic al d ru gs a m on g pa tie nt s in a gi ve n ar ea o r e nc ou nt er ed in a g iv en h ea lth ca re p ro vi de r o rg an iz at io n R eh ab ili ta tio n fa ci lit y O M R SE :0 00 00 10 6 A fa ci lit y to a ss is t i n ph ys ic al o r a dd ic tio n re co ve ry D ru g or a lc oh ol re ha bi lit at io n fa ci lit y B C IO :0 26 01 8 A re ha bi lit at io n fa ci lit y to a ss is t t he re co ve ry o f pe op le w ith d ru g or a lc oh ol a dd ic tio n. Ps yc hi at ric fa ci lit y N C IT :C 53 53 6 A p la ce d es ig ne d an d st af fe d to h ou se a nd tre at in di vi du al s th at n ee d as si st an ce w ith m en ta l d ys fu nc tio ns C om m un ity he al th ca re fa ci lit y B C IO :0 26 01 9 A c lin ic p ro vi di ng h ea lth ca re s er vi ce s to pe op le in a c er ta in a re a. Ex am pl e: p ol yc lin ic C om m un ity ou tp at ie nt c lin ic fa ci lit y B C IO :0 26 02 0 A h ea lth ca re fa ci lit y to tr ea t p at ie nt s in th e co m m un ity w ith ou t t he m s ta yi ng o ve rn ig ht . D en tis t f ac ili ty B C IO :0 26 02 1 A h ea lth ca re fa ci lit y w he re d en ta l h ea lth ca re is pr ov id ed . Ed uc at io na l f ac ili ty B C IO :0 26 02 2 A fa ci lit y in w hi ch fo rm al e du ca tio n is p ro vi de d to a s tu de nt p op ul at io n. Ea rly y ea rs fa ci lit y B C IO :0 26 02 3 A n ed uc at io na l f ac ili ty in w hi ch p re -s ch oo l ed uc at io n is p ro vi de d. Ex am pl e: n ur se ry s ch oo l Sc ho ol fa ci lit y O M R SE :0 00 00 06 4 A fa ci lit y th at is ru n by a s ch oo l o rg an iz at io n an d is th e be ar er o f a s ch oo l f un ct io n Pr im ar y sc ho ol B C IO :0 26 02 4 A s ch oo l f ac ili ty fo r y ou ng er c hi ld re n, ty pi ca lly ag ed b et w ee n fiv e an d el ev en M id dl e sc ho ol B C IO :0 26 02 5 A s ch oo l f ac ili ty p ro vi di ng e du ca tio n be tw ee n pr im ar y an d se co nd ar y sc ho ol Se co nd ar y sc ho ol B C IO :0 26 02 6 A s ch oo l f ac ili ty fo r o ld er c hi ld re n an d te en ag er s, ty pi ca lly a ge d be tw ee n el ev en a nd ei gh te en Ex am pl e: h ig h sc ho ol Page 10 of 18 Wellcome Open Research 2020, 5:124 Last updated: 11 SEP 2020 Up pe r-L ev el Su b- Le v el 1 Su b- Le v el 2 Su b- Le v el 3 Su b- Le v el 4 Su b- Le v el 5 D efi ni tio n Vo ca tio na l f ac ili ty B C IO :0 26 02 7 A n ed uc at io na l f ac ili ty p ro vi di ng p ra ct ic al ly ba se d, o cc up at io na lly -s pe ci fic te ac hi ng . U ni ve rs ity fa ci lit y B C IO :0 26 02 8 A n ed uc at io na l f ac ili ty in w hi ch s tu de nt s st ud y fo r d eg re es a nd a ca de m ic re se ar ch is d on e. C om m un ity fa ci lit y B C IO :0 26 02 9 A fa ci lit y us ed b y a gr ou p of p eo pl e liv in g in th e sa m e pl ac e or h av in g a pa rt ic ul ar ch ar ac te ris tic in c om m on . Ex am pl e: fo od b an k, re cy cl in g ce nt re Sp or t a nd e xe rc is e fa ci lit y B C IO :0 26 03 0 A c om m un ity fa ci lit y us ed fo r e xe rc is in g. Ex am pl e: g ym , s ta di um , t en ni s co ur ts , sw im m in g po ol So ci al c en tre o r C om m un ity H al l fa ci lit y B C IO :0 26 03 1 A c om m un ity fa ci lit y us ed fo r s oc ia lis in g by th os e liv in g in a g iv en a re a. Ex am pl e: Y M C A o r w or ki ng m en ’s c lu b Li br ar y fa ci lit y B C IO :0 26 03 2 A c om m un ity fa ci lit y co nt ai ni ng a c ol le ct io n of bo ok s an d le ar ni ng re so ur ce s fo r l oa n. R el ig io us fa ci lit y B C IO :0 26 03 3 A c om m un ity fa ci lit y w he re in di vi du al s or a gr ou p of p eo pl e co m e to p er fo rm a ct s of de vo tio n an d ve ne ra tio n. Ex am pl e: m os qu e, c hu rc h, te m pl e H os pi ta lit y an d ca te rin g fa ci lit y B C IO :0 26 03 4 A c om m un ity fa ci lit y us ed to s er ve fo od . Ex am pl e: d in er , r es ta ur an t, pu b or b ar A rt s an d en te rt ai nm en t fa ci lit y B C IO :0 26 03 5 A c om m un ity fa ci lit y de si gn ed to e nt er ta in o r am us e. Ex am pl e: c in em a, th ea tre , d is co R et ai l f ac ili ty B C IO :0 26 03 6 A fa ci lit y us ed a s an o ut le t f or s ho pp in g. Ex am pl e: s up er m ar ke t, m ar ke t o r s ho pp in g ce nt re R es ea rc h fa ci lit y EN VO :0 00 00 46 9 A fa ci lit y, p er m an en t o r t em po ra ry , o n la nd , i n ai r, sp ac e or w at er , w he re s ci en tifi c re se ar ch o r m ea su re m en ts c an b e un de rt ak en Ex am pl e: re se ar ch la b O ffi ce fa ci lit y B C IO :0 26 03 7 A fa ci lit y of a ro om , s et o f r oo m s, o r b ui ld in g us ed a s a pl ac e fo r c om m er ci al , p ro fe ss io na l, or b ur ea uc ra tic w or k C rim in al ju st ic e Fa ci lit y B C IO :0 26 03 8 A fa ci lit y w he re in di vi du al s ar e be in g re pr im an de d, d et ai ne d or im pr is on ed Ex am pl e: p ris on Fa ct or y fa ci lit y B C IO :0 26 03 9 A fa ci lit y of a b ui ld in g or g ro up o f b ui ld in gs w he re g oo ds a re m an uf ac tu re d or a ss em bl ed ch ie fly b y m ac hi ne Page 11 of 18 Wellcome Open Research 2020, 5:124 Last updated: 11 SEP 2020 Up pe r-L ev el Su b- Le v el 1 Su b- Le v el 2 Su b- Le v el 3 Su b- Le v el 4 Su b- Le v el 5 D efi ni tio n M ili ta ry fa ci lit y B C IO :0 26 04 0 A fa ci lit y re la tin g to o r c ha ra ct er is tic o f s ol di er s or a rm ed fo rc es . Ex am pl e: a rm y, n av y, a ir fo rc e Tr an sp or ta tio n N C IT _C 14 12 86 M et ho ds o f t ra ve lin g fro m o ne p la ce to an ot he r. Pu bl ic tr an sp or ta tio n N C IT :C 14 12 87 Fo rm s of tr an sp or ta tio n th at ru n on fi xe d ro ut es an d ar e av ai la bl e to th e pu bl ic , u su al ly fo r a se t f ar e e. g bu s, tr ai n, p la ne Pr iv at e tra ns po rt at io n B C IO :0 26 04 1 A fo rm o f t ra ns po rt at io n ow ne d by a n in di vi du al fo r i nd iv id ua l o r g ro up u se . Ex am pl e: c ar , b ic yc le , m ot or bi ke M ob ile in te rv en tio n ve nu e B C IO :0 26 04 2 A fo rm o f t ra ns po rt at io n de liv er in g in te rv en tio ns in tr an si en t l oc at io ns . Ex am pl e: m ob ile v an A m bu la nc e B C IO :0 26 04 3 A fo rm o f t ra ns po rt at io n w hi ch c an tr an sp or t pa tie nt s fo r h ea lth tr ea tm en t, an d in s om e in st an ce s w ill a ls o pr ov id e ou t-o f-h os pi ta l he al th ca re to th e pa tie nt . O ut do or e nv iro nm en t B C IO :0 26 04 4 A s ite w hi ch is a n ou td oo r l oc at io n ou ts id e of a bu ild in g. Pa rk EN VO :0 00 00 56 2 A b ou nd ed a re a of la nd , o r w at er , u su al ly in it s na tu ra l o r s em i-n at ur al (l an ds ca pe d) s ta te a nd se t a si de fo r s om e pu rp os e, u su al ly to d o w ith re cr ea tio n or c on se rv at io n Fo re st EN VO :0 00 00 11 1 A n ar ea w ith a h ig h de ns ity o f t re es . A s m al l fo re st m ay b e ca lle d a w oo d B ea ch EN VO :0 00 00 09 1 A la nd fo rm c on si st in g of lo os e ro ck p ar tic le s su ch a s sa nd , g ra ve l, sh in gl e, p eb bl es , co bb le , o r e ve n sh el l f ra gm en ts a lo ng th e sh or el in e of a b od y of w at er W at er B C IO :0 26 04 5 A n ou td oo r e nv iro nm en t s et in a n ex pa ns e of w at er . G ra ss la nd EN VO :0 00 00 10 6 A n ar ea in w hi ch g ra ss es (G ra m in ae ) a re a si gn ifi ca nt c om po ne nt o f t he v eg et at io n R oa d EN VO :0 00 00 06 4 A n op en w ay fo r t he p as sa ge o f v eh ic le s, pe rs on s, o r a ni m al s on la nd Pa th o r p av em en t B C IO :0 26 04 6 A n ou td oo r e nv iro nm en t f or th e pa ss ag e of pe rs on s or c yc lis ts o n la nd . Pa th o r p av em en t fo r p ed es tri an s B C IO :0 26 04 7 A p at h or p av em en t f or th e pa ss ag e of pe rs on s on ly o n la nd . Pa th o r p av em en t fo r c yc lis ts B C IO :0 26 04 8 A p at h or p av em en t f or th e pa ss ag e of p eo pl e us in g bi cy cl es o nl y on la nd . So ci al s et tin g (B C IO :0 29 00 0) A n ag gr eg at e of p eo pl e w ith w ho m a B C I po pu la tio n in te ra ct s. Page 12 of 18 Wellcome Open Research 2020, 5:124 Last updated: 11 SEP 2020 styles and biases of individual ontology development teams. A strength of this study is the use of an explicit, standardised, tried and tested method for ontology development created within the Human Behaviour-Change Project for a range of ontologies (Wright et al., 2020). This process incorporates international expert stakeholder feedback, as has also occurred in other related projects e.g. BCTTv1, Michie et al., 2015; Linking BCTs and Mechanisms of Action, Carey et al., 2019; TIPPME, Hollands et al., 2017; MAGI framework, Borek et al., 2019. Another strength is the integration of existing terms from other ontolo- gies where they exist, preventing duplication of entities within the wider ontology space (Norris et al., 2019). The use of entity IDs for each entity in the ontology provides a machine- readable identifier for integration in future systems and also allows interoperability between existing ontologies. The Intervention Setting Ontology has been found to be use- ful to manually annotate a large body of published intervention evaluation reports (Michie et al., 2017). These manual annota- tions are informing the development and testing of informa- tion extraction algorithms (Ganguly et al., 2018) to automate the process of identifying and organising knowledge about interventions within published reports (Michie et al., 2020). This corpus of manually and automatically extracted data on intervention setting characteristics is being made available as it is produced on the Human Behaviour-Change Project’s GitHub page. As machine-readable representations of knowl- edge, these ontologies provide a framework for applying Artifi- cial Intelligence to synthesising and interpreting evidence e.g. by identifying patterns of data organised by the BCIO. Reason- ing algorithms allow real-time up-to-date evidence synthesis that can be used to answer variants of the “big question” of behav- iour change: “What works, compared with what, for what behav- iours, how well, for how long, with whom, in what setting, and why?”, across a wide range of contexts (Michie et al., 2017). This body of work has the potential to have far-reaching use by and implications for policy-makers, practitioners and research- ers, for example, by informing evidence-based guidelines, extrapolating knowledge to under-researched populations and settings, and identifying knowledge gaps. A limitation of this work is that the intervention reports anno- tated within the ontology development mainly addressed two health-related behaviours, smoking cessation and physical activ- ity. This was due to the ontology being developed within the Human Behaviour-Change Project, which is using smoking ces- sation and physical activity interventions as initial use cases (Michie et al., 2017). However, external inter-rater reliability was tested across diverse behaviours and found to be accept- able. Future application of the ontology to a wider collection of behaviours and contexts will help extend and improve it. Conclusions The Intervention Setting Ontology provides a classification system that can be used reliably to specify the characteristics of settings where interventions take place. It will contribute to the improvement of research reporting and replication, enabling easier evidence synthesis across studies. The ontology can be used within computational tools to speed up the accumulation, interpretation and application of knowledge, such as the Knowledge System being developed within the Human Behaviour-Change Project (Michie et al., 2017). The Intervention Setting Ontology is intended to act as a foundation from which future research can build, as an ongoing and collaborative process. The ontology will allow us to increase our understanding of the settings in which interventions are implemented and how effectiveness varies across settings. Ethics Ethical approval was granted by University College London’s ethics committee (CEHP/2016/555). Data availability Underlying data The BCIO is available from: https://github.com/HumanBehaviour- ChangeProject/ontologies Archived ontology as at time of publication: https://doi.org/10.5281/ zenodo.3824323 (Norris et al., 2020). License: CC-BY 4.0 Extended data Open Science Framework: Human Behaviour-Change Project, https://doi.org/10.17605/OSF.IO/UXWDB (West et al., 2020) This project contains the following extended data: - Papers used in development of the Intervention Setting Ontology.pdf (Papers used across stages of development of the Intervention Setting Ontology, with the systematic reviews that they were identified from; https://osf.io/4qcby/) - Version 1 – Intervention Setting Ontology.pdf (Initial prototype version of Intervention Setting Ontology https://osf.io/g8qfv/) - Setting Expert Feedback Survey.pdf (Full survey provided to behavioural science and public health experts in review of the Intervention Setting Ontology; https://osf.io/8audy/) - Expert Feedback on Intervention Setting Ontology.pdf (Raw feedback received from behavioural science and public health experts; https://osf.io/npsy7/) - Intervention Setting Ontology Coding Guidelines. pdf (Manual for coding using the Intervention Setting Ontology; https://osf.io/76jty/) Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0). Page 13 of 18 Wellcome Open Research 2020, 5:124 Last updated: 11 SEP 2020 Code used to calculate alpha for IRR: https://github.com/Human- BehaviourChangeProject/Automation-InterRater-Reliability. Archived code as at time of publication: https://doi.org/10.5281/ zenodo.3833816 (Finnerty & Moore, 2020). License: GNU General Public License v3.0 Acknowledgements We are grateful to Danielle D’Lima and Gill Forbes for annotat- ing papers for inter-rater reliability testing, and to Olga Perski and Paul Chadwick for providing comments on an earlier draft of this paper. References Antoniou G, van Harmelen F: Web Ontology Language: OWL. In: S. S. & S. R. (Eds.), Handbook on Ontologies. International Handbooks on Information Systems. Berlin: Springer. 2004; 67–92. Publisher Full Text Arp R, Smith B, Spear AD: Building Ontologies with Basic Formal Ontology. Massachusetts: MIT Press. 2015. Reference Source Ashburner M, Ball CA, Blake JA, et al.: Gene ontology: tool for the unification of biology. Nat Genet. 2000; 25(1): 25–29. PubMed Abstract | Publisher Full Text | Free Full Text Balhoff JP, Brush MH, Christopherson L, et al.: Tailoring the NCI Thesaurus for Use in The OBO Library. International Conference of Biomedical Ontologies. 2017. Reference Source Black N, Eisma M, Viechtbauer W, et al.: Systematic review and meta-analysis of control groups in smoking cessation trials: Implications for conducting and interpreting systematic reviews. Addiction. 2020. Publisher Full Text Borek AJ, Smith JR, Greaves CJ, et al.: Developing and applying a framework to understand mechanisms of action in group-based, behaviour change interventions: the MAGI mixed-methods study. Efficacy and Mechanism Evaluation. 2019; 6(3). PubMed Abstract | Publisher Full Text Buttigieg PL, Morrison N, Smith B, et al.: The environment ontology: contextualising biological and biomedical entities. J Biomed Semantics. 2013; 4(1): 43. PubMed Abstract | Publisher Full Text | Free Full Text Carey RN, Connell LE, Johnston M, et al.: Behavior change techniques and their mechanisms of action: a synthesis of links described in published intervention literature. Ann Behav Med. 2019; 53(8): 693–707. PubMed Abstract | Publisher Full Text | Free Full Text Cohen J: A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960; 20(1): 37–46. Publisher Full Text De Bruin M, Viechtbauer W, Eisma MC, et al.: Identifying effective behavioural components of Intervention and Comparison group support provided in SMOKing cEssation (IC-SMOKE) interventions: a systematic review protocol. Syst Rev. 2016; 5(1): 77. PubMed Abstract | Publisher Full Text | Free Full Text Finnerty A, Moore C: HumanBehaviourChangeProject/Automation-InterRater- Reliability: Release of HBCP inter-rater reliability code v1.0.0 (Version v1.0.0). Zenodo. 2020. http://www.doi.org/10.5281/zenodo.3833816 Finnerty AN, Wright AJ, Norris E, et al.: Development of an Intervention Population Ontology for behaviour change interventions. Wellcome Open Res. In preparation. Ganguly D, Deleris LA, Mac Aonghusa P, et al.: Unsupervised information extraction from behaviour change literature. Stud Health Technol Inform. 2018; 247: 680–684. PubMed Abstract Gwet KL: Handbook of Inter-Rater Reliability: The definitive guide to measuring the extent of agreement among raters. Advanced Analytics, LLC. 2014. Hastings J: Primer on Ontologies. Methods Mol Biol. 2017; 1446: 3–13. PubMed Abstract | Publisher Full Text Hayes AF, Krippendorff K: Answering the Call for a Standard Reliability Measure for Coding Data. Communication Methods and Measures. 2007; 1(1): 77–89. Publisher Full Text He Y, Xiang Z, Zheng J, et al.: The eXtensible ontology development (XOD) principles and tool implementation to support ontology interoperability. J Biomed Semantics. 2018; 9(1): 3. Publisher Full Text Hicks A, Hanna J, Welch D, et al.: The ontology of medically related social entities: recent developments. J Biomed Semantics. 2016; 7: 47. PubMed Abstract | Publisher Full Text | Free Full Text Hoffmann TC, Glasziou PP, Boutron I, et al.: Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014; 348: g1687. Publisher Full Text Hollands GJ, Bignardi G, Johnston M, et al.: The TIPPME intervention typology for changing environments to change behaviour. Nat Hum Behav. 2017; 1(8): 1–9. Publisher Full Text Jackson RC, Balhoff JP, Douglass E, et al.: ROBOT: A Tool for Automating Ontology Workflows. BMC Bioinformatics. 2019; 20(1): 407. PubMed Abstract | Publisher Full Text | Free Full Text Larsen KR, Michie S, Hekler EB, et al.: Behavior change interventions: the potential of ontologies for advancing science and practice. J Behav Med. 2017; 40(1): 6-22. PubMed Abstract | Publisher Full Text Marques MM, Carey RN, Evans F, et al.: Delivering Behavior Change Interventions: Development of a Mode of Delivery Ontology. Wellcome Open Res. 2020. Publisher Full Text Michal K, Michal Š, Zdeněk B: Interoperability through ontologies. IFAC Proceedings Volumes. 2012; 45(7): 196–200. Publisher Full Text Michie S, Johnston M: Optimising the value of the evidence generated in implementation science: the use of ontologies to address the challenges. Implement Sci. 2017; 12(1): 131. PubMed Abstract | Publisher Full Text | Free Full Text Michie S, Thomas J, Johnston M, et al.: The Human Behaviour-Change Project: harnessing the power of artificial intelligence and machine learning for evidence synthesis and interpretation. Implement Sci. 2017; 12(1): 121. PubMed Abstract | Publisher Full Text | Free Full Text Michie S, Thomas J, Mac Aonghusa P, et al.: The Human Behaviour-Change Project: An Artificial Intelligence System to answer questions about changing behaviour. Wellcome Open Res. 2020. Publisher Full Text Michie S, West R, Finnerty AN, et al.: Representation of behaviour change interventions and their evaluation: Development of the Upper Level of the Behaviour Change Intervention Ontology. Wellcome Open Res. 2020. Publisher Full Text Michie S, West R, Hastings J: Creating ontological definitions for use in science. Qeios. 2019. Publisher Full Text Michie S, Wood CE, Johnston M, et al.: Behaviour change techniques: the development and evaluation of a taxonomic method for reporting and describing behaviour change interventions (a suite of five studies involving consensus methods, randomised controlled trials and analysis of qualitative data). Health Technol Assess. 2015; 19(99): 1–188. PubMed Abstract | Publisher Full Text | Free Full Text Montgomery P, Grant S, Mayo-Wilson E, et al.: Reporting randomised trials of social and psychological interventions: the CONSORT-SPI 2018 Extension. Trials. 2018; 19(1): 407. PubMed Abstract | Publisher Full Text | Free Full Text Norris E, Finnerty AN, Hastings J, et al.: A scoping review of ontologies related to human behaviour change. Nat Hum Behav. 2019; 3(2): 164–172. PubMed Abstract | Publisher Full Text Norris, Hastings, Finnerty: HumanBehaviourChangeProject/ontologies: Upper- Level, Setting & MoD papers Submitted (Version v1.0). Zenodo. 2020. http://www.doi.org/10.5281/zenodo.3824323 Schulz KF, Altman DG, Moher D: CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMC Med. 2010; 8(1): 18. PubMed Abstract | Publisher Full Text | Free Full Text Page 14 of 18 Wellcome Open Research 2020, 5:124 Last updated: 11 SEP 2020 Shindo H, Munesada Y, Matsumoto Y: PDFAnno: a web-based linguistic annotation tool for pdf documents. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). 2018. Reference Source Smith B, Ashburner M, Rosse C, et al.: The OBO Foundry: coordinated evolution of ontologies to support biomedical data integration. Nat Biotechnol. 2007; 25(11): 1251–1255. PubMed Abstract | Publisher Full Text | Free Full Text Smith B, Ceusters W, Klagges B, et al.: Relations in biomedical ontologies. Genome Biol. 2005; 6(5): R46. PubMed Abstract | Publisher Full Text | Free Full Text Thomas J, Brunton J: EPPI-Reviewer 4.0: software for research synthesis. London: Institute of Education. 2010. Reference Source UK Government: Types of School. 2019. Reference Source West R, Michie S, Shawe-Taylor J, et al.: Human Behaviour-Change Project. 2020. http://www.doi.org/10.17605/OSF.IO/UXWDB Wright AJ, Norris E, Finnerty AN, et al.: Ontologies relevant to behaviour change interventions: A method for their development. Wellcome Open Res. 2020. Publisher Full Text Page 15 of 18 Wellcome Open Research 2020, 5:124 Last updated: 11 SEP 2020 Open Peer Review Current Peer Review Status: Version 1 Reviewer Report 11 September 2020 https://doi.org/10.21956/wellcomeopenres.17444.r40321 © 2020 Noone C. This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Chris Noone School of Psychology, National University of Ireland Galway, Galway, Ireland This article reports the development of an intervention setting ontology for behaviour change research. The rationale for this work is strong as the field would clearly benefit from the availability of a shared language for discussing where behaviour change interventions take place. This paper reports this development process very clearly and the underlying data is easy to navigate and understand. I do agree with my fellow reviewer that the article would be easier to follow if the results for each stage were reported after the methods for that stage are detailed. This would be analogous to the structure of multi-study articles. One important aspect of this project that could be explained in more detail is how the research community can contribute to the ongoing development of this ontology using the GitHub repository. For example, the term "developing country" is contested and the apparent conflation (if I have interpreted the onotology correctly) of low income countries and low income areas within countries, might be issues that researchers would like to provide feedback on, but many are not familiar with GitHub. Perhaps a guide on providing feedback could be developed and placed in the OSFproject associated with this article? Is the work clearly and accurately presented and does it cite the current literature? Yes Is the study design appropriate and is the work technically sound? Yes Are sufficient details of methods and analysis provided to allow replication by others? Yes If applicable, is the statistical analysis and its interpretation appropriate?   Page 16 of 18 Wellcome Open Research 2020, 5:124 Last updated: 11 SEP 2020 Yes Are all the source data underlying the results available to ensure full reproducibility? Yes Are the conclusions drawn adequately supported by the results? Yes Competing Interests: I'm on the EHPS Open Science Special Interest Group steering committee that is chaired by Dr Norris. Reviewer Expertise: Health Psychology I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Reviewer Report 24 June 2020 https://doi.org/10.21956/wellcomeopenres.17444.r39190 © 2020 Epton T. This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Tracy Epton Division of Psychology and Mental Health, School of Health Sciences, University of Manchester, Manchester, UK The manuscript describes the development of an ontology for an intervention setting. It provides a useful tool   There are a few minor issues that need addressing.   Methods: Step 2: For the preliminary ontology, how were the initial labels and definitions formulated once they had been extracted from the papers (bottom up approach) and existing ontologies (top down approach) e.g., process; number of people involved? ○ Results: Step 2: why weren’t changes made to the entities that had a low kappa to improve the reliability of rating?   ○ Step 4: what fields were the behavioural scientists from?○   Discussion: Could a link to the GitHub portal be added in the text on p.7 so people can easily provide feedback on the ontology? ○   Page 17 of 18 Wellcome Open Research 2020, 5:124 Last updated: 11 SEP 2020   Generally: I found myself continuously going back and forth between methods and results for each step. I wonder if, in this instance, it is more appropriate to put the methods and results for each step together for ease of reading (i.e., like the reporting for a series of studies). ○ Is the work clearly and accurately presented and does it cite the current literature? Yes Is the study design appropriate and is the work technically sound? Yes Are sufficient details of methods and analysis provided to allow replication by others? Yes If applicable, is the statistical analysis and its interpretation appropriate? Not applicable Are all the source data underlying the results available to ensure full reproducibility? Yes Are the conclusions drawn adequately supported by the results? Yes Competing Interests: I have been involved as a volunteer on various behaviour change projects Reviewer Expertise: Health behaviour change I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.   Page 18 of 18 Wellcome Open Research 2020, 5:124 Last updated: 11 SEP 2020