The retrieval of documents containing the contents of a protocol design is one method of determining a clinical research protocol. The document containing the guidelines covers the overall information on clinical research protocols. This is the most basic approach for gathering information to develop a protocol. Chan et al. [3] proposed SPIRIT, which is a high-quality guideline containing 33 checklist items for the development of a clinical trial protocol. Meeker-O’Connell et al. [26] developed a principle document that defines the factors needed to assure patient safety and reliability in a trial. Moreover, some guidelines specify protocols for examining the efficacy of food or food components for specific diseases [27]. For instance, documents describing gut health and immunity, diet-related cancer, and atherosclerosis are included [28–30]. However, guideline-based approaches use subjective judgment in determining the information to be included [31]. This limitation can result in different outcomes depending on the user.

Database-based information retrieval systems can be utilized to retrieve clinical protocols. Zarin et al. developed clinicaltrials.gov, the largest database retrieval system, by collecting all published clinical trial documents, including regulatory mandates and a broad group of trial sponsors [18]. Tasneem et al. established and operated a relational database containing all clinical trials registered with clinicaltrials.gov [16]. Furthermore, systems for protocol retrieval use general document retrieval technologies, e.g., PubMed, Scopus, Web of Science, and Google Scholar [32–35]. However, current database-based retrieval systems have limited ability for protocol-specific search objectives, such as retrieving the protocol structure or sequentially selecting context-dependent protocol elements.

Intelligent systems are an effective approach for retrieving clinical trial protocols. Tsatsaronis et al. developed an intelligent system based on a context-aware approach for automated protocol design [15]. Their system supports study- and domain-driven searches. Study-driven searches use the parameters (i.e., condition, intervention) of a particular trial as provided by a researcher. In domain-driven searches, a researcher selects options according to the study domain; the system then automatically searches and categorizes the retrieved information. However, a system such as this is currently not accessible. We assume that the author terminated the system.

A clinical trial protocol presents the structure of a clinical trial and is composed of various elements that can be clustered into key factors. In this study, we defined five key factors based on a previous baseline research project [25]: design, subject, variables, statistical issues, and descriptions. The design factor determines how the trial is structured and modeled to measure data generated during the trial. The subject factor determines who is eligible to participate in the trial and how they are treated to ensure the generalizability of the target population. The variables are the parameters to be measured to evaluate the efficacy or safety of a drug or treatment. The statistical issues describe how the clinical trial will be analyzed, specifying sampling procedures or statistical significance. Finally, the description factor covers additional information such as the organization, different phases, and additional explanations of the protocol or trial itself.

We selected and clustered elements from Aggregate Analysis of ClinicalTrials.gov (AACT), which was released on March 27, 2015 [16]. To do this, we first downloaded a dump file of the AACT database and completely overhauled the loaded database to obtain 42 tables of 270 columns. Then, we classified the data types into four elements: categorical, value, description, and not union (N/U). Categorical-type elements contain categorical variables; the sequential selection of these elements can determine the protocol structure (S1 File). Value-type elements include interval and ratio data that are important values in key factors. Description-type elements contain additional explanatory text, numeric values, and abbreviated words or dates for the description factor. The N/U-type element consists of primary keys, foreign keys, and database management values.

We clustered the categorical and value types into the pre-defined five key factors according to the above-mentioned criteria, including design, subject, variable, and statistical issue factors (Table 1).

We designed a table schema and amassed the data compilation progress. The N/U elements were eliminated because we constructed a relational table with key-value attributes, discarding unnecessary keys and values for database management. The use of key-value attributes makes it possible to search the skeleton structure of a clinical trial protocol efficiently and effectively manage the data storage for deploying inconsistent data [36]. We designed a table schema accounting for these aspects (Table 2). The next step was data compilation. We organized the element values by resolving typographical errors and reflecting dependency structures. Then, we removed some control characters and type conversions. Furthermore, we removed ambiguous design types, which are null, and expanded the access and observations (patient registry) to search for specific clinical trial protocols. As a result, we collected 184,634 clinical trial protocols and their detailed information. The resulting database can be used to optimize query refinement for retrieving protocol information.

Although we developed a clinical trial protocol database by using a frame structure for all protocols to have a similar structure, frame structure similarity does not guarantee similarity of the detailed protocol content. MeSH offers a potential solution and is used for indexing and cataloging clinical trials in ClinicalTrials.gov and AACT [16, 27]. However, MeSH has limited coverage that does not extend across the spectrum of various biomedical terminologies [37]. To solve this limitation, various biomedical semantic features were extracted to find or filter similar clinical trials in the structure being searched.

We generated filterable semantic features related to the conditions and interventions that are considered significant in clinical trials, resulting in a subdivided semantic similarity search. The condition was a phenotype, including any diseases and disorders, observed during clinical trials as well as reported symptoms. The disease-specific phenotype is a set of observable characteristics. Drugs commonly refer to interventions that are the focus of clinical trials; they can involve chemical compounds [38]. Similar clinical trials can be searched for or filtered using each of the corresponding elements. In addition, the identification of similar target genes or proteins is a potential method of searching for similarities among chemical compounds and phenotypes as they are a molecular proxy that links them [39, 40]. Thus, we applied named entity recognition (NER) to the phenotypes, chemical compounds, and genes to enable a semantic search, for which semantic filters were employed for the following description elements: brief title, official title, brief summary, detailed description, keywords, and conditions (Fig 2).

We appended gene entities in the elements of semantic filters. The gene annotation tool, Moara, was used for gene NER, considering that Moara can perform both recognition and normalization of gene entities, recognizing entities and their positions in the input text, and linking the entities to gene IDs in a known gene database [47]. Moara provides various preconstructed machine learning models for certain organism species. For our task, we adopted the human-oriented model. For gene normalization, we obtained lists of gene IDs corresponding to each gene entity. The gene ID with the highest score was selected and mapped to the recognized gene term.

The structures of clinical trial protocols have become increasingly complex [48]. The complexity of the protocol level is inversely related to clinical trial performance, as complex protocols negatively impact factors such as protocol amendment rates, patient recruitment, and retention rates [49]. In addition, the increasing complexity of the protocols hinders the design of new protocols because clinicians referring to previous clinical trials to design a protocol inevitably face difficulties in searching for suitable examples. Thus, from a query refinement standpoint, we developed the CLIPS web application to provide a graph querying interface for retrieving information about reliable clinical trial protocols, rather than a text querying interface that cannot visualize the dependency among prior elements affecting protocol structure [50, 51].

We defined categorical-type elements as the frame structures of protocols. Although we have provided default orders of the elements, the user is free to choose the order. Once a decision about the order has been made, the user can find varying combinations of protocol elements via the graph-based search interface. Dependent elements are retrieved from the database in real-time, and the user can confirm the number of existing protocols corresponding to selected elements. The user also can search for other protocol frame structures after clipping the selected structure to the user clip pane. To search for complete information about the selected protocol structures, the user must click on a clipped protocol to examine the details of the selected protocol and trial information. Furthermore, the user can add semantic filters when searching for complete information to reduce the search scope or focus on certain areas of clinical trials. The data flow of the interface is illustrated in Fig 3.

The backend of the interface was developed using Node.js [52], and the visualization of the interface was manipulated by d3.js [53]. We developed custom functions on d3.js to present each element title and protocol count for the user’s selection. To implement semantic filtering of the user’s free-text input, a backend engine was connected to the representational state transfer (REST) NER API. This provided NER of processed entities and types, allowing relevant entities to be searched for in the database. We designed the system architecture to combine the interface application, APIs, and database for stable operation in a cloud-computing environment (S1 Fig).

We collected 184,634 clinical trial protocols and the frame structures of 13,210 clinical trial protocols from which we extracted 5,765,054 phenotypes, 1,151,053 chemical compounds, and 222,966 gene features for semantic filtering (Table 4). Furthermore, we designed a continuous process of data update so that the protocol methods could evolve naturally, thus enhancing the quality of the database (S2 Fig). In conclusion, we developed a database system that efficiently retrieves information about existing clinical trial protocols for use in designing new clinical trial protocols.

The web application provides a service for retrieving protocol structures and inquiring about protocol information. The user traverses four stages when using this service: (1) setting the order of the protocols; (2) designing the protocol structure by selecting the elements that correspond to each sequence; (3) setting various functions required to search for the desired protocol information; and (4) providing the desired protocol information to explore the contents in detail. We developed the necessary interfaces to perform all of these processes (Fig 4).

Before retrieving the protocol structure, the user needs to define the protocol sequence. This process uses a drag-and-drop interface to sort the list into one box and set the order; it allows users to work more intuitively [54]. After determining the protocol sequence, the vector-based collapsible and zoomable tree diagram visualization interface, which is an uncomplicated tool, is used for navigating the protocol structures with the selected elements [55, 56]. The loaded protocol data are assembled into a hierarchical data structure. This visualization is rendered as a relation tree with a parent/child structure. The user clicks on the edge of the tree to add the next-step protocol. Conversely, to remove protocol edges from the current stage, the user must click on the parent edge of the previous step. The entire data structure is synchronized and updated every time the process occurs [57].

After the protocol structure has been retrieved, the user obtains protocol information based on the selected protocol structure. We developed a function called Clip to back up the protocol design. This allows the user to reuse previously selected protocol structures and receive corresponding study information. In addition, a protocol-information-filter function allows the user to retrieve the study information. The user searches for a disease and generates a label that contains the disease code. The protocol information is then filtered according to the set label. The resulting data are rendered as a table, which can be sorted with respect to the column entities to focus on and export specific data. When exploring detailed protocol information, our system transforms the data into a collapsible interface instead of providing raw text.

Although protocol design concepts have evolved globally, the development of tools to design clinical trial protocols is trivial [58–60]. Our aim was to simplify clinical trial retrieval and the design stage by developing a dedicated interface. We expect this to be the starting point for the creation, sharing, and development of more clinical trial protocols.

The goal of CLIPS was to provide a database system for information retrieval method that can search complex clinical trial protocols. To achieve this, we developed a search tool that can build and utilize a database suitable for protocol structures. Furthermore, we created semantic features in CLIPS using text-mining methods. As a result, it was possible to perform accurate searches using the protocol’s contextual meaning. To evaluate the performance of CLIPS, we attempted to verify whether the semantic filters perform better than a keyword search.

For technically validating the CLIPS’ semantic filter we used the relational information between clinical trial protocols and corresponding disease conditions as collected from clinicaltrials.gov [18]. As this disease condition assignment is manually curated by experts and does not originate from the protocol itself, it can be used as a gold standard to evaluate the semantic filters of CLIPS.

The gold standard set of disease conditions and corresponding trial protocols was obtained by crawling the topic page of clinicaltrials.gov [18]. Among 25 conditional categories provided by clinicaltrials.gov, the “Cancers and Other Neoplasms” condition category was selected as the first set because it covers 44.74% of the total protocol set. Consequently, the corresponding trial protocols and corresponding disease conditions were identified. For instance, in our gold standard set, 353 distinct protocols were associated with the disease condition “Abdominal neoplasm.” As a result, a set of 82,584 distinct protocols corresponding to 520 disease conditions was compiled (as of July 12, 2017) and used as a gold standard set for technical validation. In addition, we expanded the gold standard set to the rest of the 24 categories for sufficient validation. The expanded set has 289,956 distinct protocols corresponding to 6,172 disease conditions that were compiled (as of June 21, 2020). However, the collected set showed a 2.4% protocol loss from clinicaltrials.gov. This was occasioned by server overload during data crawling, and we omitted the loss.

The semantic search performance of CLIPS was validated using the following procedure. In CLIPS, search keywords that contained the disease condition’s nomenclature were supplied as input queries to the system. The semantic entities were translated from the search keyword through the text-mining-based models described in the previous section. The results were obtained by conducting a search using the translated semantic entities from the CLIPS database. Exact matching with the AACT database was used as a baseline. CLIPS and AACT databases were configured on a single local server.

We used a condition name (e.g., Adrenocortical Carcinoma) as a search keyword to retrieve the condition field of the source database and semantic entities (e.g., C0206686) from CLIPS. We then validated our retrievals by comparing the number of retrieved identifiers (NCTID) of protocols (S2 File) [61] and calculated the precision, recall, and F1-score of the results. Precision refers to how accurately the model is categorized and is calculated as by the ratio of properly categorized data to the total data (1). Recall is the number of positively classified data divided by the original number of positive data (2). The F1-Score is a harmonic mean that considers the complementary characteristics of precision and recall (3); it is commonly used to compare the performance of different information retrieval systems [62]. True positive + false positive is the count of all NCTIDs retrieved from each database. True positive is the count of the retrieved NCTIDs of each database intersected with the gold standard. True positive+ false negative is the total number of NCTIDs in the gold standard corresponding to each input condition name.

The CLIPS’ F1-Score of CLIPS (0.515) was higher than that of the keyword search (0.38) (Fig 5A). The precision of CLIPS was 0.437, which was slightly lower than that of the keyword search (0.668), but it outperformed the keyword search by more than a factor of two in terms of recall (0.63 and 0.26, respectively). In the expanded conditional categories, the F1-Score of CLIPS (0.55) was higher than that of the keyword search (0.12) (Fig 5B). The precision of CLIPS was 0.44, which was slightly higher than that of the keyword search (0.39), but it also outperformed the keyword search by more than a factor of two in terms of recall (0.38 and 0.08, respectively). As higher recall values are a positive factor in clinical trial design, CLIPS can retrieve more protocols that provide more suitable references for protocol design.

As described earlier, the CLIPS search system was developed for different purposes than those of the existing clinical trial search systems. CLIPS is intended to assist in the effective design of a specific trial protocol, whereas existing search systems are generally used to process a variety of information on clinical trials. Therefore, to evaluate the performance of CLIPS, an evaluation method that reflects this purpose should be constructed. As the ultimate purpose of a search system is to help users collect the information they require, user’s satisfaction is a significant if subjective, measure of performance. Thus, the evaluation of a search system should be able to quantify the subjective impressions of users as well as objective indicators. By considering these factors, we conducted an evaluation trial that compared CLIPS with the conventional search system provided by clinicaltrials.gov.

Ten participants aged between 24 and 33 were recruited from a group of experts in the field of bioinformatics (Bio and Brain Engineering Department of Korea Advanced Institute of Science and Technology, Republic of Korea) They included two undergraduates, three master’s students, four Ph.D. candidates, and one postdoctoral researcher. All participants provided informed consent before conducting the evaluation trial. The participants were assigned the task of finding a suitable clinical trial set for a simulated problem. Two separate tasks were given to the participants, who were asked to construct the most common previous protocol design under the given trial conditions and research questions and perform each task using each of the two search systems within the time limit (5 min each). To construct the most common trial design, participants had to collect information about the various elements of the trial protocol (Fig 7). After the task, participants completed a questionnaire on their subjective satisfaction regarding the system. The questionnaire consisted of six questions that inquired about the participants’ satisfaction using a 7-point Likert scale [63].

Participants were observed to perform better when using CLIPS than when using the clinicaltrials.gov search system. By using CLIPS, participants obtained more answers within the time limit, and the average time required to perform the task was shorter than that with clinicaltrials.gov. The number of clinical trials retrieved from CLIPS was less than that from clinicaltrials.gov because it was possible to apply more detailed search filters to narrow the search scope. Participants were more satisfied with CLIPS than with the existing search systems, as evidenced by the average score of the questionnaire responses (Table 5). These results show that CLIPS can be effectively used to retrieve certain types of trial protocols.