Movember is a leading charity changing the face of men's health. Their mission is to discover what works for men - and what does not. They see that men's health is in crisis as men are dying on average 4.5 years earlier than women, and for largely preventable reasons.
Orygen believes that all young people deserve to grow into adulthood with optimal mental health. They are an outcomes-focused organisation, like Movember. As most mental health disorders begin between the early teens to the mid-20s they believe in treating early and focusing on recovery. In Australia one in five young people will have experienced a depressive episode by the time they turn 18. Three times more males than females are likely to die from suicide in Australia.
The problem
Movember recognises that many male clients have complex needs and require gender-sensitive engagement in psychotherapy, and that often practitioners are unaware of ‘best-practice' when it comes to engaging male clients. To ensure the delivery of best-practice for male clients, especially younger men, Movember funded a partnership with Orygen to develop a world-first online training program aimed at improving therapists' capacity to engage and connect with their male clients.
Being evidence based, results-oriented organisations, Movember and Orygen wished to evaluate the effectiveness of their innovative training for practising therapists via a randomised controlled trial.
Movember had initially used a 'build-your-own-survey' product for the pilot, and quickly realised that this was not a core skill set they needed to or could rapidly develop to fulfil their mission. Movember decided they needed to find a strategic partner. They needed a company that could steer the design and technology integration of the survey with a third-party online learning management system (LMS), collect audio responses from participants as well as quantitative feedback about the effectiveness of the training, and manage a complex workflow of pre-, post-training and application follow-up surveys for control and study participants. They realised they also needed a strategic partner that would help them achieve the positive results they wanted not only more quickly but more cost-effectively by integrating all these elements seamlessly. Their concern was whether they could find one organisation that could do all this.
The solution
Logicly's WebSurvey Research was chosen, not only for our IT technical competence, but also because of our previous experience with logistically complex research projects, such as the evaluation of the Link-me trial, and our background in research methodology.
Mapping the workflow
After an initial briefing, Movember was able to hand the planning and implementation of the surveys, LMS, and data management over to our team. Logicly's WebSurvey Research team began by building a comprehensive survey workflow based on the research protocol prepared by the Movember research team. We mapped out each step in the process from inviting practising therapists to participate in the online training program, to the timing of reminders, the placement of objective tests and at which points, regretfully, participants became ineligible for study continuance. As further planning occurred, any changes to the initial workflow were tracked and confirmed by Movember, so that Movember was able to sign off on a final design that would ensure a robust, high quality data collection that would deliver the data they needed. They felt confident and ready for the next stage due to this quality assurance process.
Movember's confidence was rewarded and the design proved robust and high quality. There was a seamless transition from the survey platform into the learning management system for the practising therapists, and although a help desk had been set up as a safety net, and was useful for eligibility questions, there were no help desk calls related to difficulties accessing either the survey or the LMS. As a result of this ease of engagement there was high retention and satisfaction rates among the practising therapists, which in turn ensured higher quality data.
Specialised technical expertise
Once we had mapped out the survey and training workflow, we needed to undertake the more difficult implementation stage. Implementation meant taking our existing survey modules and writing custom code to program the timing, out of study rules and custom emails. To achieve this, Logicly's WebSurvey Research team worked directly with the LMS provider, OpenLMS, to integrate the training system with the online survey. We worked with their existing APIs to automatically create user accounts for participants when they had successfully finished the pre-training survey, saving the busy practising therapists the time-wasting hassle of creating new log-ins. We also used APIs to make “calls” to the LMS to track the practising therapists' progress, so that the survey we designed for Movember could send appropriate reminder emails based on training completion status. This work required the specialised technical expertise of WebSurvey's developers as the LMS did not have these capabilities straight out of the box. We had to work closely with OpenLMS to understand their system enough to modify our code to work seamlessly with theirs. These modifications managed participant progress automatically and saved significant amounts of the Movember research team's time. The workflow was tested multiple times across a wide variety of browsers and operating systems and their different versions. We do this for all aspects of our online design.
Stakeholder sensitive design
In an office environment where IT support ensures all computers use the same browser and operating environment, versions problems can be easily anticipated and solved. In the browser and operating environment smorgasbord of the real online world a full-service help desk always needs to be provided as part of risk and quick recovery assurance. As is often the case, some of the practising therapists needed help to allow their browsers to record and playback their audio responses, and to play the scenario videos. Our team was able to solve any browser and operating system problems. Help desks limit frustration, enhance engagement and improve the quality of the results findings.
It was also important to Movember that this study was available to a wide range of practising therapists. Movember's focus is on knowledge accessibility. As a strategic partner we also focus, for ethical and other reasons, on online accessibility. One area of our expertise is in providing advice to research teams in implementation decisions that both meet their research and ethical protocols and are technically doable. For example, scale questions presented in a grid are not accessible to people using a screen reader - so while grids look neat and compact and are widely used, WebSurvey does not use them unless an informed decision has been made. We present scale questions so that each question, and the scale, can be read easily by screen readers. The 'build-your-own-survey' product Movember had used for the pilot could only use grids, and when Movember asked if we could use a grid, we investigated whether screen reader technology had changed enough that it would now be accessible. It had not, so to maintain the accessibility Movember desired and to prevent unintended access problems WebSurvey and Movember elected not to use a grid.
A third focus of accessibility was for the Movember research team. To improve the efficiency of the research we used the pre-existing researchers' study codebook to reduce the data preparation required by the statistician. This enabled the statistician to take the returned data and load it into the analytical tool without having to map survey output variables to survey analysis variables. Statistician time was reduced, and interim and final results were able to be created faster.
Ensuring study success
Since Movember's survey involved pre-, post-training, and follow-up surveys their research team was very concerned there would be the traditional large decrease in response rates over the three stages. This is even more likely to occur when the final follow-up survey is many weeks or months after the post-training survey. Movember knew that the quality of the research and strength of the analysis was dependent on the adherence to the research protocols around timing of follow-up surveys. For the data collected to be useful both the control and the intervention groups had to stay on a similar schedule so that differences in survey timing was not a potential factor in the analysis. To address this key research risk, we worked with Movember to plan a series of additional reminder emails and personalised time lapsed targeted extensions that met the research protocols. The final response rates for the follow-up survey exceeded requirements; a key outcome of this complex research had been met and Movember had response rates to meet their statistical requirements.
At the end of the project Movember reflected that the WebSurvey Research team was “knowledgeable and competent and at the cutting edge of creating solutions for customers. It was all coded up and integrated (with the LMS) for us, rather than having to do it ourselves. It's been a delight.”
Customer delight is our aim.
PostScript
Three months after our involvement in the project ended the WebSurvey Research Director received an email from one of Movember's lead researchers that made our day:
“In case you did not hear the trial crushed it due to the heavy lifting from you and the team”
