In the Netherlands, people on sick leave for two years can apply for a work incapacity pension from the Dutch Social Security Institute (SSI) on the basis of the Work and Income Act depending on work capacity (WIA). People who receive a WIA incapacity for work pension and who still have (partial) work capacity are supposed to earn part of their income. To achieve this objective, these people benefit from support from the SSI in their return to work. This support is offered by labor experts. During a face-to-face interview, the work expert assesses the obstacles to overcome and the virtual reality interventions that facilitate the return to work for a particular person. Based on this interview, a rehabilitation plan is developed and the person with partial work capacity is referred to a corresponding rehabilitation provider to receive VR intervention. The labor expert monitors the progress of the return to work.
In this study, we used a stepwise modified Delphi technique  by combining the Delphi technique [11, 12] and the noun group technique  obtain a multidisciplinary consensus from our panel of experts. The panel was made up of labor experts, social workers and insurance doctors. The Delphi consisted of several cycles using online questionnaires and an online meeting that used the nominal group technique. The nominal group technique was used to structure the meeting in order to explore the differences between individual labor experts and social workers in their opinion on return to work barriers and interventions and to reach consensus among participants. through group discussions. . Previous studies have successfully used the combination of the Delphi technique and the nominal cluster technique [14, 15].
At the start of the Delphi study, we compiled a list of potential return-to-work factors and virtual reality interventions by searching scientific studies and semi-scientific (grey) literature on related populations and talking with professionals. We have retrieved 58 RTW factors from semi-scientific studies [16, 17]scientific journals [18,19,20,21], and conversations with professionals working at ISS. Virtual reality interventions were retrieved through an analysis of interventions currently offered to long-term disabled workers in the Netherlands. These were complemented by virtual reality interventions found in [22,23,24,25,26,27,28] and semi-scientific  Literature. To search the academic literature for virtual reality interventions, we used the following definition: “an evidence-based, multi-professional approach that is provided in different settings, services and activities to people of working age with disabilities, health-related limitations or restrictions with functioning at work, and whose primary objective is to optimize work participation” .
Our expert panel included people who had worked for at least one year as a labor expert, social worker or insurance doctor with the RTW for long-term disabled workers at the Dutch SSI.
We recruited the panel of labor experts and stakeholders via a recruitment message published in the SSI’s internal newsletter. From the responses to the recruitment message, we selected representative labor experts and social workers from different regions, ages, genders and years of experience. In addition, we recruited insurance doctors experienced in the rehabilitation of long-term patients by e-mail.
Data collection and analysis
The Delphi study consisted of three sub-questions which the experts answered during five rounds of Delphi questionnaires. An overview of these rounds and consensus rules can be found in Table 1.
Factors associated with the return to work of workers with long-term disabilities (cycles 1, 2, 3)
In the first question, the experts were asked to what extent they agreed with the statement “This factor influences the return to work of long-term disabled workers”. Experts could rate the extent of their agreement on a 5-point Likert scale. The response options were: (1) totally disagree, (2) disagree, (3) neutral, (4) agree and (5) totally agree. Consensus has been reached if ≥ 80% of experts rated (4) agree or (5) strongly agree for a particular factor. These factors were accepted without further discussion. If the percentage of consensus was between ≥ 70% and < 80%, the RT factor had to be noted again by the experts during the next round of the questionnaire. If the percentage of consensus was less than 70%, the factor was not considered important for the return to work of workers on long-term disability and was excluded from the study. Experts were also given the opportunity to add factors that they felt were important for this group's return to work and that were not on the list. These new factors were added in the second round, excluding duplicates. In the second round, the experts were again asked to indicate the extent to which they agreed with the newly added factors and the factors for which no consensus had been reached in the previous round. Per factor, the experts received an overview of the number of participants who chose a specific answer category for this question in the previous round and their own answer. Then, the percentage of consensus was calculated in the same way as in the first round. Factors for which the percentage of consensus was between ≥ 70% and < 80% after the second round were scored again in the third round.
Factors that can be targeted by virtual reality interventions (Rounds 1, 2, 3)
In the second question, the experts were asked if they thought that a factor could be targeted by a return to work intervention. Response options were: (1) yes or (2) no. A consensus was established if ≥ 80% of the experts obtained (1) yes or (2) no. These factors were accepted without further discussion. Factors that had a consensus percentage < 70% were considered untargetable with virtual reality interventions. As for the first question, if the percentage of consensus was between ≥ 70% and < 80%, the RAT factor was scored again by the experts in the following questionnaire. In the second and third rounds, the experts received an overview of the number of participants who chose a specific answer category for this question in the previous round and their own answer. In the third round, the newly added factors for which there was no agreement reached in the second round were once again marked by the experts.
Effective Virtual Reality Interventions to Target Return-to-Work Factors (Cycles 4 and 5)
Finally, factors that had been determined by consensus to be (a) associated with RTW of long-term disabled workers and (b) potentially targetable using VR interventions were again presented to the experts in a fourth round. Experts were asked to rate how effective 22 groups of virtual reality interventions would be in targeting each of the factors. These groups of virtual reality interventions have been collected from the (academic) literature and from practice. The experts were given a brief description of the virtual reality interventions and were able to indicate their response on a 3-point Likert scale. Response options were: (1) not effective, (2) somewhat effective, and (3) very effective. After this round, consensus percentages were calculated for each intervention on each factor. Virtual reality interventions for which ≥ 70% of experts rated (3) highly effective were determined to be targetable with RAT factors and were accepted without further discussion. Virtual reality interventions for which ≥ 50% and < 70% of experts rated the intervention as (3) very effective were re-presented to experts for scoring in the next round. Virtual reality interventions for which <50% rated the intervention (3) very effective were excluded from subsequent Delphi cycles.
The fifth round consisted of an online meeting and an online questionnaire. Due to a combination of the large number of virtual reality interventions for factors for which consensus was not reached in the fourth round, the large number of potential interventions, and time constraints, only a selection of factors and their virtual reality interventions could be discussed and noted. again. We selected the 8 factors with the highest percentage of consensus for the statement: “this factor is associated with the return to work of workers with long-term disabilities”, to reach a consensus on the most effective VR interventions for target these factors for VR interventions for which consensus was not yet reached.
VR interventions for the other factors were not scored again in the fifth round. The results of the fourth round were therefore the final results for the VR interventions of these RTW factors. RAT factors for which ≥ 70% of experts scored (5) strongly agree that the factor is important for the RAT, and for which there was consensus that this factor could be influenced by an intervention of RV (round 2) were carried to the fifth round. Participants received an overview of the percentage of participants who chose a certain answer option in the previous round.
During the online meeting, each factor was presented on screen and, using an online voting tool, participants rated all virtual reality interventions (for which no consensus had been reached). reached in the fourth round) depending on whether they were (1) not effective or (2) somewhat/very effective in targeting the factor. Then the results were discussed with all the participants. The results of this discussion were incorporated into the final online questionnaire. Only half of the participants were able to attend this meeting. The other participants received the questions on paper.
In the final online questionnaire, all participants (including those who were not present during the online meeting) were asked to re-evaluate all virtual reality interventions (for which no consensus had been reached in the fourth round) on a dichotomous scale indicating whether they were effective or not. to target each of the eight RAT factors. Consensus in this round was reached if ≥ 70% of experts agreed that the intervention was (2) somewhat/very effective in targeting a particular factor. These VR interventions were accepted without further discussion.