Development and Validation of a Radiotherapy Administration Checklist for Patients Undergoing Radiation Therapy

Abstract

Background and objective: Radiotherapy, the use of high-energy ionizing radiation to treat diseases, is increasingly employed in cancer treatment due to technological advancements. Precision and competency are paramount in radiotherapy administration, necessitating tools to standardize care and minimize medical errors. This study aimed to develop and validate a radiotherapy administration checklist to serve as a standardized guide for the procedure.

Materials and methods: A methodological research design was employed to develop the radiotherapy administration checklist. The development process involved five phases: (i) preliminary phase (literature review, focus group discussions, assessment of current practices, item pool generation, and preliminary draft preparation), (ii) validation phase (content validity assessment using the modified Delphi technique and construct validation through factor analysis), (iii) pilot testing, (iv) final trial and reliability assessment (internal consistency and inter-rater reliability), and (v) evaluation phase.

Results: The final radiotherapy administration checklist comprised 29 items and demonstrated strong validity and reliability. Content validity scores (S-CVI/Avg. and S-CVI/UA) were 0.97 and 0.79, respectively. Cronbach's alpha (0.64) and inter-rater reliability (0.76) were within acceptable ranges. Factor analysis revealed that all 29 items loaded onto eight distinct factors.

Conclusion: The developed radiotherapy administration checklist is a valid and reliable tool with psychometric properties within the expected range. It can be used as a standardized tool to ensure safe, uniform, and optimal delivery of radiotherapy.

Introduction

Non-communicable diseases are rising by the day worldwide, and cancer is one of the diseases with significantly increasing disease burden. According to GLOBOCAN data 2020, over 50 million people are having cancer diagnosis since 5 years, 20% of people develop cancer during the lifetime, 1 in 8 men and 1 in 11 women die from cancer [1]. Countries with middle and low income per capita have higher incidence, morbidity and mortality associated with cancer [2]. Asia accounts for majority of cancer related mortality, 58.3% of worldwide cancer deaths in 2020 were recorded in Asia [3]. Case mortality rate is higher in developing countries due to lack of public awareness, and lack of treatment resources. Radiotherapy remains one of the main components in treatment of cancer, and it requires substantial infrastructure, initial training and investment. Radiotherapy is a modality of treatment by the use of ionizing radiation to damage the DNA of cancer cells, to reduce the chances of reoccurrence of cancer, and to alleviate the symptoms. Radiotherapy can be given independently or in combinations with other cancer treatment modalities, such as chemotherapy and surgery. Radiotherapy is given to approximately 50% of all cancer patients during the course of their treatment [4]. Radiation can be delivered in two ways to the location of the cancer. External beam radiation is delivered from outside the body by aiming the ionizing radiation to the location of tumour. Internal radiation or brachytherapy is delivered by internally placed sources, sealed in catheters/ seeds directly in the tumour [5].

With the development of modern technologies in big cities and upper middle income countries, the process of delivery of radiotherapy has been made available and more efficient, but in low income countries due to lack of resources, 90% of patients requiring radiotherapy lack the access to radiotherapy treatment [6]. Treatment facilities are available in well established institutes of India, but there is a lack of uniformity in standards of the treatment delivery due to the lack of audits, checklists and standard guidelines for radiotherapy, it compromises the process of delivery of care, increases the chances of error in radiation delivery, reduces the satisfaction of patient, and it can lead to compromised outcome of treatment. To our knowledge no such tool is available in the Indian setting. Therefore, we developed a Radiotherapy administration Checklist consisting of 29 items under 3 domains: pre- procedure, and intra procedure, post- procedure. The radiotherapy administration Checklist aims at implementing uniformity of care, reduce the occurrence of errors, and increasing the overall effectiveness of radiotherapy procedure.

Materials and Methods

In the present study quantitative research approach and methodological research design has been used with the objective to develop Radiotherapy administration Checklist for patients undergoing radiotherapy. The study was conducted at Radiotherapy unit of All India Institute of Medical Sciences (AIIMS), Jodhpur, Rajasthan. The sample consisted of 300 patients undergoing Radiotherapy (200 Teletherapy and 100 Brachytherapy) at AIIMS, Jodhpur selected by non probability convenience sampling (Event sampling). Sample size was decided after tool development based on the total number of items in checklist (29 items), 10 patients per item were taken (290 patients), which was rounded off to 300. Brachytherapy sample was smaller due to less availability of patients undergoing brachytherapy. Data Was collected from September to November 2021.

Ethical clearance was obtained from Institutional Ethical Committee of AIIMS, Jodhpur, Rajasthan (AIIMS/ IEC/2021/3601). Written informed consent was obtained from the subjects after providing adequate information, and explanation of the study.

Data collection tool

Radiotherapy administration Checklist developed in five phases : i) the preliminary phase, ii) the validation phase, iii) pilot testing, iv) final try out, v) evaluation phase.

Phase I: Preliminary phase

Preliminary draft of Radiotherapy administration Checklist (Appendix: I) was developed by; detailed review of literature, assessment of current practices, focused group discussion, and generation of item pool based on the conclusions drawn from the aforementioned resources.

The preliminary draft of Radiotherapy administration Checklist was developed with 34 items under following thee domains (pre procedure, intra procedure, and post procedure), and 10 sub domains were formulated: patient and family information and consent, comprehensive assessment, treatment planning, treatment delivery (teletherapy), patient set up (teletherapy), treatment delivery procedures (teletherapy), treatment delivery (brachytherapy), immediate interventions, recording and reporting, follow up. The scoring key was developed in the terms of: 1 for Yes, 0 for No, and 0 for Not applicable. Maximum score for checklist was 34 and minimum score was 0.

Phase II: Validation phase

This phase included content validity of Radiotherapy administration Checklist by using modified Delphi technique, the preliminary draft of Radiotherapy administration Checklist was sent to 15 experts with the letter requesting experts for content validity and content validity performa. A panel of 10 experts provided their consent to provide expert guidance in order to reach the final consensus. Three rounds of modified Delphi were conducted to establish the content validity within acceptable range.

Modifications done after first round of modified Delphi

The domains remained same whereas two sub domains ‘treatment delivery procedures (teletherapy)’ and ‘recording and reporting’ were removed, the sub domain ‘ patient set up’ was stated before ‘treatment delivery’ sub domain, and one sub domain ‘immediate interventions’ changed to ‘post procedure interventions’. The statements with overlapping meaning with another statement were modified and merged and some new statements added in second draft. Items in the ‘patient and family information and consent’, ‘treatment planning’, patient set up’, ‘immediate interventions’ and ‘follow up’ sub domains were added as per the experts suggestions and the need of modifications. Items in the ‘comprehensive assessment’, ‘patient set up’,’ treatment delivery (teletherapy)’, were omitted as per the experts guidance.

Second draft of Radiotherapy administration Checklist

After compiling the correction and suggestion from experts under modified Delphi rounds, Second draft of Radiotherapy administration Checklist was prepared with 38 items under 3 domains: Pre procedure (17 items), Intra procedure (11 items), Post procedure (10 items) and 8 sub domains; patient and family information and consent (4 items), comprehensive assessment (4 items), treatment planning (6 items), patient set up (3 items), treatment delivery (teletherapy) (3 items), treatment delivery (brachytherapy) (8 items), post procedure interventions (7 items), follow up (3 items). Scoring of 1 has been given for ‘YES’, 0 will be given for ‘NO’ and for ‘NA’ (not applicable).

Modifications done after first round of modified Delphi

The sub domains of second draft of Radiotherapy administration Checklist were merged into three main domains ‘Pre procedure’, ‘Intra procedure’, and ‘Post procedure’ as per the suggestions of the experts. Some items from the second draft were deleted in the third draft as they were overlapping with other items and there was ambiguity of meaning in some items. As per expert’s suggestion some items were reframed. Items in some of the domain were overlapping with other items and were not feasible for the assessment were deleted from the checklist. Nine items were deleted, and one item was merged with other suitable item with some modifications. One more item was added as per the experts opinion and need of the checklist.

Third/ Final draft of Radiotherapy administration Checklist

After compiling the corrections and suggestions from the experts under modified Delphi rounds, modifications were done and third/ final draft of Radiotherapy administration Checklist prepared, which is again sent to all 10 experts among them all 10 experts replied. No major changes were suggested in the third round and hence all of the 29 items were retained in third draft with 3 domains: pre procedure (12 items), intra procedure (11 items), post procedure (6 items). The minimum score of the checklist is 0 and maximum score is 29. The content validity, S-CVI/Avg. and S-CVI/UA were 0.97 and 0.79 respectively.

Phase III: Pilot study

Pilot study was done in September 2021. Radiotherapy administration checklist was used as the tool for data collection. The checklist was implemented on 1/10^th of total sample (30 patients undergoing radiotherapy) for data collection. Pilot study revealed that the Radiotherapy administration Checklist was comprehensible, easy to implement, and feasible with adequate composition and sequence of items in the checklist.

Phase IV: Final try out of the tool

Final draft of the Radiotherapy administration Checklist was tried out on 300 Patients undergoing radiotherapy (200 teletherapy and 100 brachytherapy) at AIIMS, Jodhpur. The data collection was done in the months of September to November 2020. The average completion time of the checklist was 8 – 10 minutes for teletherapy and as per the need of procedure in brachytherapy.

Content Validity

Content validity of the Radiotherapy administration Checklist was calculated by evaluation of the checklist by the panel of 10 experts. Evaluation of checklist was done through Content validity performa which was developed by Davis in 1992 under the 4 relevancy criteria for each item in checklist: highly relevant (4), quite relevant (3), somewhat relevant (1), and not relevant (0). Scoring was done by dichotomizing these four criteria in relevant which includes highly relevant and quite relevant; and not relevant includes somewhat relevant and not relevant. On the basis of 10 expert’s evaluation, content validity index (CVI) was calculated for the items (I-CVI) and for the checklist (S-CVI). The I-CVI ranged from 0.8 to 1 and S-CVI/Ave is 0.97.

Construct Validity

Data was analyzed using SPSS (version 20.0). Construct validity of Radiotherapy administration Checklist was established by exploratory factor analysis. Factor extraction condenses items into a smaller number of factors and is used to identify the number of underlying dimensions [7]. Principal component analysis with varimax rotation was applied for factor extraction and it yielded a total of 8 factors with eigenvalues >1(between 1.00 to 14.55), factor 1 had eigenvalue 14.55, factor 2 had the eigenvalue 1.48, factor 3 had eigenvalue 1.25, factor 4 had eigenvalue 1.20, factor 5 had eigenvalue 1.13, factor 6 had eigenvalue 1.11, factor 7 had eigenvalue 1.08, factor 8 had eigenvalue 1.00. The variance explained by item I, II, III, IV, V, VI, VII, VIII were 50.18, 5.13, 4.33, 4.14, 3.90, 3.83, 3.75, 3.45 respectively. The 8 factors accounted for 78.74% of total variance. All the items loaded under 8 factors, so all items were retained in the checklist (Table 1).

Table 1. Total Variance Explained by Items.

Component		Initial Eigenvalues			Extraction Sums of Squared Loadings			Rotation Sums of Squared Loadings
	Total	% of Variance	Cumulative %	Total	% of Variance	Cumulative %	Total	% of Variance	Cumulative %
1	14.55	50.18	50.18	14.55	50.18	50.18	13.69	47.22	47.22
2	1.48	5.13	55.32	1.48	5.13	55.32	2.25	7.76	54.99
3	1.25	4.33	59.65	1.25	4.33	59.65	1.26	4.34	59.34
4	1.2	4.14	63.8	1.2	4.14	63.8	1.17	4.05	63.39
5	1.13	3.9	67.7	1.13	3.9	67.7	1.15	3.97	67.37
6	1.11	3.83	71.53	1.11	3.83	71.53	1.11	3.84	71.21
7	1.08	3.75	75.29	1.08	3.75	75.29	1.1	3.79	75.01
8	1	3.45	78.74	1	3.45	78.74	1.08	3.72	78.74
9	0.94	3.27	82.01
10	0.9	3.13	85.14
11	0.86	2.97	88.12
12	0.77	2.67	90.79
13	0.6	2.07	92.87
14	0.37	1.27	94.15
15	0.3	1.06	95.21
16	0.26	0.9	96.11
17	0.22	0.77	96.88
18	0.16	0.57	97.45
19	0.15	0.52	97.98
20	0.13	0.45	98.44
21	0.08	0.28	98.73
22	0.07	0.27	99
23	0.07	0.25	99.26
24	0.06	0.2	99.46
25	0.05	0.17	99.64
26	0.04	0.14	99.79
27	0.03	0.12	99.92
28	0.01	0.06	99.98
29	0	0.01	100

Principal component analysis

Rotated component matrix

For analyzing rotated component matrix, extraction method of principal component analysis applied along with varimax rotation with Kaiser normalization. Table 2 represents the rotated component matrix; items 2,11,17,18,19, 20,21,22,23 and 27 are correlated with component 1 with correlation values 0.63, 0.91, 0.96, 0.96, 0.95, 0.95, 0.98, 0.96, 0.96, 0.84; items 6, 25 and 28 were correlated with component 2 with correlation values 0.71, 0.72, 0.62; items 4, 10, 15 were correlated with component 3 with correlation values 0.79, 0.75, 0.01; items 9, 16, 26 were correlated with component 4 with correlation values 0.79, 0.01, 0.59; items 1, 8 were correlated with component 5 with correlation values 0.68, 0.65; items 7, 13 were correlated with component 6 with correlation values 0.60, 0.04; item 3 was correlated with component 7 with correlation value of 0.87, and items 5,12,14, 24, 29 were correlated with component 8 with correlation values 0.53, 0.02, 0.03, 0.14 and 0.76 respectively.

Table 2. Rotated Component Matrix.

Items				Component
	1	2	3	4	5	6	7	8
1					0.68
2	0.63
3							0.87
4			0.79
5								0.53
6		0.71
7						0.6
8					0.65
9				0.79
10			0.72
11	0.91
12								0.02
13						0.04
14								0.03
15			0.01
16				0.01
17	0.96
18	0.96
19	0.95
20	0.95
21	0.98
22	0.96
23	0.96
24								0.14
25		0.72
26				0.59
27	0.84
28		0.62
29								0.76

Extraction method: Principal component analysis; Rotation method: Varimax rotation with Kaiser normalization.

Scree plot

A scree plot is a graphical method used in the selection of the number of relevant components or factors to be considered in a principal components analysis or a factor analysis. Conceptually, the scree plot is a way of visualizing the magnitude of the variability associated with each one of the components extracted in a principal component analysis.

Figure 1 depicts that initial 8 factors had a major contribution to the total variance. Factor 1 had eigenvalue 14.55, factor 2 had the eigenvalue 1.48, factor 3 had eigenvalue 1.25, factor 4 had eigenvalue 1.20, factor 5 had eigenvalue 1.13, factor 6 had eigenvalue 1.11, factor 7 had eigenvalue 1.08, factor 8 had eigenvalue 1.00. Successive to first 8 factors, the scree plot curve does not have any further deflection and is smoother (Figure 1).

Figure 1. Scree Plot of Radiotherapy Administration Checklist.

Reliability of the Radiotherapy administration Checklist

Internal Consistency

There were a total of 29 items in the Radiotherapy administration Checklist and overall Cronbach’s alpha of Radiotherapy administration Checklist was found to be 0.65 which indicates good internal consistency.

Cronbach’s alpha if item deleted

When the individual item was deleted the value of Cronbach’s alpha either remained same or decreased for 24 items which indicated that those 24 items are contributing to the reliability of the checklist. But for 5 items (Item 12,13,14,15,16) the value of Cronbach’s alpha was increased when the individual item was deleted. The Cronbach’s alpha values were increased for 5 items (Item 12,13,14,15,16) because all these items are only applicable for teletherapy procedure and at one time only one procedure was observed (either teletherapy or brachytherapy), As the items has significant contribution in the checklist despite the increased value of Cronbach’s alpha after their deletion. Hence, all 29 items were retained in the checklist after consultation with guide and experts (Table 3).

Table 3. Reliability Analysis of Checklist by Cronbach’s alpha.

Items	Scale Mean if Item Deleted	Scale Variance if Item Deleted	Corrected Item-Total Correlation	Cronbach's Alpha if Item Deleted
1	17.71	12.06	-0.01	0.65
2	18.22	9.44	0.77	0.56
3	17.74	12.08	-0.03	0.65
4	17.71	12.03	0.01	0.65
5	17.73	12.11	-0.04	0.65
6	18.19	9.4	0.79	0.56
7	17.71	12.03	0.02	0.64
8	17.71	12.05	0	0.65
9	17.74	12.01	0.01	0.65
10	17.7	12.08	-0.02	0.65
11	18.34	9.55	0.78	0.57
12	18.02	14.66	-0.76	0.73
13	18.05	15.03	-0.84	0.74
14	18.03	15.1	-0.87	0.74
15	18.01	15.23	-0.91	0.74
16	18.05	15.03	-0.85	0.74
17	18.36	9.44	0.84	0.56
18	18.36	9.43	0.84	0.56
19	18.36	9.48	0.83	0.56
20	18.36	9.45	0.84	0.56
21	18.35	9.35	0.86	0.56
22	18.36	9.4	0.85	0.56
23	18.36	9.41	0.85	0.56
24	17.73	12.08	-0.03	0.65
25	18.16	9.52	0.74	0.57
26	17.69	12.06	0.01	0.64
27	18.34	9.78	0.7	0.58
28	18.2	9.46	0.76	0.56
29	17.74	11.93	0.06	0.64

Interrater Reliability

In the present study interrater reliability was established for a sample of 50 patients: 34 teletherapy and 16 brachytherapy. Cohen’s kappa was calculated which is a measure of interrater reliability and was found to be 0.77 for the Radiotherapy administration Checklist that implies 77% as a percentage of agreement between two independent observers.

Phase V: Evaluation phase

After completing four phases of tool development, Radiotherapy administration Checklist consisting of 29 items was finalized. The psychometric properties (reliability and validity) of the checklist are within expected range. Internal consistency of the checklist was analyzed by Cronbach’s alpha which was found 0.65, indicating good internal consistency of the checklist. For equivalence, interrater reliability was calculated by Cohen’s kappa and was found to be 0.77 for the Radiotherapy administration checklist signified that percentage of agreement was 77%. Content validity index was calculated for content validity which was 0.97, indicating good content validity of the checklist. For calculation of construct validity of checklist, principal component factor analysis was applied which generated 8 factors. Most of the items were having loading value >.01 on factors which shows the Radiotherapy administration checklist was having good construct validity. All 8 factors accounted for 78.74% of the variance. Total 29 items is in the checklist. Maximum score is 29 and minimum score is 0. The scoring was; 1 for yes and score of 0 for no, and not applicable.

Discussion

Radiotherapy administration is an area of treatment delivery which requires great clinical precision and attention to detail to ensure optimal and hazard free delivery of care and to achieve optimal patient outcomes. In present study, a Radiotherapy administration Checklist for patients undergoing Radiotherapy was developed under three domains namely: Pre procedure, Intra procedure, and Post procedure. Similarly, Ogama N. and Ogama N. used methodological design to develop an oral assessment tool to evaluate appetite in patients with head and neck cancer receiving radiotherapy, the oral assessment tool was developed under three domains namely: Dysgeusia, Xerostomia and Oral mucositis [8].Likewise, Langegard U. et. al. developed a radiotherapy- related symptom assessment scale, this tool was developed under 13 variables including: fatigue, insomnia, pain, loss of appetite, dyspnoea, cognitive impairment, worry, anxiety, nausea, sadness, constipation, diarrhoea, skin reaction for patients with brain tumors undergoing proton beam therapy [9].

Furthermore, the Content validity index was evaluated as 0.97, which indicates good content validity of the checklist. Polit and Beck concluded that for the scale to have excellent content validity SCI/Ave should be 0.90 or higher [10].

For establishing the construct validity of Radiotherapy administration Checklist principle component factor analysis was applied which yielded 8 factors according to the components with total 78.74% variance explained. These findings are in correspondence with Rashvand et. al. study to develop a tool for safe nursing care assessment, for which construct validity by principal component analysis extracted four factors with overall variance of 63.54% [11].

Reliability of Radiotherapy administration Checklist was analyzed in terms of internal consistency which was found to be 0.65 by Cronbach’s alpha. Cohen’s kappa is a measure for inter – rater reliability which was 0.77 for the Radiotherapy administration Checklist, it signified that percentage of agreement was 77%. These findings are in loop with a study was conducted by Macedo and Bohomol at Patient Safety Centers in health care institutions to validate an instrument for the self - assessment for which Cronbach’s alpha was 0.857 and percentage of agreement was 70% [12]. M Knöös and M Ostman conducted a similar study to test the reliability and validity of the Oral Assessment Guide in patients receiving radiotherapy and the inter-rater reliability was >71% indicating a high concordance.30 Likewise, a previous study for the development of a risk assessment tool for the prediction of fall examined percentage of agreement between two observers which was found to be 77% and Cronbach’s alpha was 0.75 [13].

The findings of present study suggests that Radiotherapy administration Checklist for patients undergoing radiotherapy is valid and reliable tool for the implementation of uniformity of care in radiotherapy procedure, to provide standard guidelines for the procedure of radiotherapy delivery, to improve the overall care outcome, and to improve the dynamics of multidisciplinary team by involving all health care personnel.

Strength of the study

In the area of radiotherapy there is a lack of standardized tools and checklists, this checklist is is specifically meant for serving as a guide for implementing the procedure of Radiotherapy, the availability of checklist will lead to uniformity of care, and improvement of overall quality of care in radiotherapy administration procedure. The Radiotherapy administration Checklist was found to have good practicability in Radiotherapy units.

In conclusion, on the basis of findings of the present study the conclusion can be drawn that Radiotherapy administration Checklist is feasible, highly reliable and valid checklist that can be used to implement uniformity of care, and to evaluate and improve the comprehensive outcome of patients undergoing radiotherapy.

Recommendations

A study can be replicated to assess the impact and measurable outcome of Radiotherapy administration Checklist. A comparative study can be conducted to establish comparison in patient outcome by using present Radiotherapy administration Checklist vs. checklists available in Radiotherapy units. An observational study can be conducted to assess the effect of Radiotherapy administration Checklist on patient outcome.

A qualitative study can be conducted to assess the perception towards Radiotherapy administration Checklist in Radiotherapy unit.

Ethical clearance

Ethical clearance was obtained from Institutional Ethical Committee of AIIMS, Jodhpur, Rajasthan (AIIMS/ IEC/2021/3601).

Financial support

Nil

Conflict of interest

There are no conflicts of interest.