You are here
The use of EORTC measures in daily clinical practice—A synopsis of a newly developed manual
European Journal of Cancer, November 2016, Pages 73 - 81
Cancer has increasingly become a chronic condition and the routine collection of patient-reported outcomes (PROs) like quality of life is widely recommended for clinical practice. Nonetheless, the successful implementation of PROs is still a major challenge, although common barriers to and facilitators of their beneficial use are well known. To support health care professionals and other stakeholders in the implementation of the EORTC PRO measures, the EORTC Quality of Life Group provides guidance on issues considered important for their use in daily clinical practice. Herein, we present an outline of the newly developed “'Manual for the use of EORTC measures in daily clinical practice”, covering the following issues: * a rationale for using EORTC measures in routine care *selection of EORTC measures, timing of assessments, scoring and presentation of results * aspects of a strategic implementation * electronic data assessment and telemonitoring, and * further use of EORTC measures and ethical considerations. Next to an extensive overview of currently available literature, the manual specifically focuses on knowledge about EORTC measures to give evidence-based recommendations whenever possible and to encourage readers and end-users of EORTC measures to contribute to further needed high-quality research. The manual will be accessible on the EORTC Quality of Life Group website's homepage and will be periodically updated to take into account any new knowledge due to medical, technical, regulatory and scientific advances.
- Science and guidelines advocate routine patient-reported outcome (PRO) assessment in oncology.
- The EORTC Quality of Life Group is a major stakeholder responsible for developing and disseminating high-quality PRO measures.
- A newly developed Quality of Life Group manual provides guidance for successful integration of PROs into daily clinical routine.
- The manual deals with issues important for everybody who is interested/engaged in the use of PRO measures.
- Discussed issues include theoretical, practical, scientific and ethical aspects.
Keywords: EORTC Quality of Life Group, Patient-reported outcomes, Quality of life, Neoplasms, Manual, Implementation, Daily clinical routine, Guideline.
The focus of cancer care has shifted from a focus mainly on survival of patients to how patients experience their disease over the course of treatment, receive aftercare and live with cancer as a chronic condition. There is a growing demand for routine monitoring of patient-reported outcomes (PROs) like quality of life (QOL) to complement clinical data with the patient's perspective. Although routine QOL assessments are recommended in both the literature , , , , and  and evidence-based national guidelines for cancer care , , and , they are rarely incorporated in clinical practice, though common barriers and possible solutions are well known , , and .
As the EORTC instruments are some of the most often used PRO measures (PROMs) in oncology with high acceptability , , and , we developed an EORTC-specific manual, to supplement the variety of available EORTC measures with guidance according to their use in routine care. Our Manual for the use of EORTC measures in daily clinical practice (available at http://groups.eortc.be/qol/) addresses all stakeholders interested in using EORTC instruments as a tool in routine patient care, providing general information on PROs and giving particular information on the EORTC measures. Corresponding to the classic manuscript style of a narrative review, the aim was to collect an extensive body of literature, extracting the most important information, evidence-based recommendations and explicitly referencing all sources of information when cited. Furthermore, considerations for an appropriate implementation strategy and issues exceeding routine care make up a large part, to give a comprehensive picture of the potential of PRO assessments.
In 2011, the International Society for Quality of Life Research developed a first version of a User's Guide for Implementing Patient-Reported Outcomes Assessment in Clinical Practice, which was also published as a review article  and updated in 2015 , to provide basic information for clinicians who are interested in collecting any kind of PROs (e.g. patient-reported adverse events, functioning, health status and QOL) in their clinical practice. Following short informational paragraphs at the beginning of each chapter, the guide provides checklists in note form about needed resources, advantages, disadvantages and further reading references for each aspect dealt with. As the International Society for Quality of Life Research guide and the EORTC manual differ noticeably concerning covered issues and how information is presented, users might benefit most by combining the two sources.
This article is a synopsis of the newly developed EORTC manual, which offers a narrative review of relevant PRO literature of five different areas of interest: (1) Why EORTC PROMs should be used in routine clinical practice; (2) practical implications concerning selection, timing of assessments, scoring and presentation of data assessed with EORTC measures; (3) implementation of EORTC measures in daily clinical practice; (4) electronic data assessment and PRO-telemonitoring; and (5) further areas of application of EORTC measures and ethical considerations.
2. Why EORTC measures should be used in routine clinical practice
The presented manual uses the term 'EORTC measures' as an umbrella for all PROMs developed by the EORTC Quality of Life Group (QLG):
- Stand-alone questionnaires covering issues important for all cancer patients (EORTC QLQ-C30 , EORTC QLQ-C15-PAL , EORTC IN-PATSAT32 )
- Additional disease-specific modules .
- EORTC computerised adaptive testing (CAT) measures .
- Single items from the EORTC item library (former item bank)  (to create ad hoc checklists when a specific module is not available or needs to be extended)
EORTC measures undergo a rigorous development and validation procedure  and have been shown to be applicable across multiple countries and nationalities , , and , as development already assesses cross-cultural appropriateness , and high-quality translations  are available.
Several studies have provided evidence that the regular use of PROMs, including the EORTC measures, facilitate and improve communication between patients and health care professionals , , , and , as clinicians' access to PRO data makes discussion of more intimate issues more likely , , and  and increases their awareness of patients' functioning and well-being  and  without prolonging consultation time , , , , and . Furthermore, the use of PROMs has the potential to facilitate shared medical decision-making as patient participation is enhanced , to provide appropriate referral to specialists , , , and , and to enhance continuity of care . Routine electronic assessment of PRO (ePRO) data enables their being linked to medical data in the electronic health records or other medical registries which support clinicians in both therapy-associated data interpretation and scientific data processing. Efforts to systematically include PROMs have increased in recent years, including in publications and white papers providing advice and best practices , , and  and the project of the International Consortium for Health Outcomes Measurement, which is developing recommendations of common PRO data sets for different diseases, including EORTC measures , , and .
3. Practical implications concerning selection, timing of assessments, scoring and presentation of data assessed with EORTC measures
Owing to the high-methodological quality and wide range of EORTC measures, users just need to decide what kind of data they are interested in (EORTC QLQ-C30 alone, additional disease-related modules, further issues like information or satisfaction, or EORTC CAT measures). Issues like the mode of administration (e.g. paper–pencil, electronic, phone calls) and patient burden need to be taken into account, as they should impose the least possible burden on the patients, health care professionals and clinical workflow without omitting important information.
Limited evidence-based recommendations for the timing of routine assessments with EORTC measures are available and are mainly derived from expert opinions  and . In clinical practice, comprehensive knowledge of the disease and specific cancer types, treatment effects and their interaction , or at least elaborate hypotheses concerning this matter, should guide the timing of assessments (e.g. in advance decisions, if assessments should be time- or event-driven), as otherwise important information might be missed or patients' scores might be misjudged . In addition, a predefined rationale should guide the frequency of PRO assessments, as the evaluation of applied interventions, routine monitoring of in- or outpatients and long-term monitoring require different schedules to gather meaningful data. Few clinical models for determining assessment time-points and frequency have been developed  and , and further research and individual adaptation of schedules are warranted.
For EORTC measures, a scoring manual comprehensively illustrating the calculation of outcomes is available . Scores might be interpreted compared to previous assessments of the same patient, related to reference populations (e.g. cancer subgroups, general population, age or gender groups) or to thresholds for clinical importance. There is a growing interest in establishing such thresholds to improve the usefulness of the EORTC QLQ-C30 in daily clinical practice , , and , enabling the differentiation between patients with and without clinically important QOL changes, fostering its use as a screening tool and providing a reliable decision aid guiding patient-clinician communication. The EORTC QLG currently funds an ongoing cross-cultural project, developing thresholds for clinical importance for all scales of the EORTC QLQ-C30.
Available literature on graphical presentation formats for the QLQ-C30 mostly focuses on group-level data  from clinical trials. The results of two recent studies  and  investigating presentation styles for PRO results from individual patients are not conclusive enough to make definite recommendations on how PRO data should be presented in clinical practice. Presumably, health care professionals and patients may differ in their preferences regarding types of graphical presentation and the amount of information included.
Specific training and information materials tailored to patients' or health care professionals' needs should be used to enable them to correctly understand the presented results of the EORTC measures. Fig. 1 provides examples of presentation styles used by members of the EORTC QLG.
Examples of graphical presentation of QLQ-C30 results.
4. Implementation of EORTC measures in daily clinical practice
Integration of PRO assessments using EORTC measures into clinical routine is a complex health care intervention with multiple interacting components on different levels of the clinical system, all of which are sensitive to multiple influences and barriers  and . Based on available recommendations on PRO implementation into clinical practice [e.g. Refs. , , , , , and  and its own experience, the EORTC QLG proposes some practical approaches to support successful integration (please refer to Table 1).
Single steps of successful implementation of EORTC measures in daily clinical practice.
|Understand current practice before applying strategies for integration.||Existing patient care pathways, knowledge and attitudes towards EORTC measures as well as the skills of potential adopters should be assessed and resources and barriers analysed . A baseline assessment could identify elements relevant for the adaption of practice change, such as benefits, harms and costs .|
|Engage all relevant stakeholders in all stages of implementation.||This should include the introduction of existing national guidelines, providing the opportunity for health care professionals to discuss current practice , engaging senior clinicians, nurse champions and opinion leaders and assigning a coordinator throughout the implementation process .|
|Collaborate to define the goals and expectations for routine assessments with EORTC measures up front to ensure relevance.||Health care professionals' active participation in the ongoing integration process ensures that they regard assessments with EORTC measures to be relevant for their clinical practice, thereby facilitating uptake of the new intervention into clinical routine.|
|Make data actionable.||Automatic ‘flagging’ of clinically important scores, providing longitudinal interpretation of what signifies a clinically important change in EORTC measures data as well as linking of results with clinical practice guidelines and interventions , , and  are examples how EORTC measures data can be made available for health care professionals.|
|Provide training, coaching and support for health care professionals as well as patients and their informal caregivers.||A multifaceted, interactive approach , , and  is expected to be more effective for practice change than large-scale, passive education materials  and detailed reports on such specific programs are available  and . At the patient level, disease, cultural and personal aspects should be considered to reduce response burden .|
|Evaluate integration process and outcome.||Defining outcome as well as process indicators together with health care professionals will help them reflect on the process of integration, to identify obstacles and to adapt implementation strategies where necessary . Regular reviews will determine progress, elicit problems and engage health care professionals in identifying efficient strategies to overcome barriers .|
|Consider organisational context||Although implementation is not recommended concurrent with other major organisational changes , organisational adoption will be necessary to support sustainability of routine assessment of EORTC measures and their being embedded in integrated cancer care pathways.|
|Long-term evaluation of effective integration.||Long-term follow-up is necessary to determine if short-term changes persist and whether surrogate outcomes point towards benefits . Quasi-experimental, observational studies or service development and evaluation models based on quality improvement and implementation science methods might be more informative than randomised controlled trials .|
5. Electronic data assessment and telemonitoring
Routine ePRO assessment with the EORTC measures provides real-time results, can reduce time and labour needed for data entry and calculation as well as continuous expenditures (e.g. for printing, though costs for software support might cancel out these savings) provides high-data quality and improved care may be achieved. Paper–pencil versions can serve as a back-up or assessment method for those patients who would otherwise decline assessment (e.g. older people  and ), as studies report a general equivalence between paper–pencil and computer-based versions of PROMs . At present, jointly with the EORTC QLG, Mapi  is developing guidelines for graphical presentation formats of electronic versions of the EORTC measures.
Infrastructural requirements for ePRO comprise both technical and educational aspects. A comprehensive IT infrastructure is needed and includes technical devices for data collection and output (workstations, hand-held devices, preferably equipped with touchscreens), appropriate software solutions and network facilities for data transmission, storage and back-up, technical support and updates. Furthermore, user skills regarding computer literacy and open-mindedness towards PRO assessments are of special importance.
There are a variety of systems available which allow ePRO , , and , but systems vary remarkably concerning their features. In choosing software for ePRO of EORTC measures, it should be carefully checked if it is flexible enough and able to provide all desired functionalities and possibly any future necessary adoptions. Since 2009, the EORTC QLG has been supporting the development of the CHES.EORTC software , which allows ePRO in daily clinical practice, clinical studies and long-term follow-up via telemonitoring and provides a data exchange interface for clinical information systems based on the Health Level Seven standard as well as an elaborated graphical results presentation of patient-level data . Using CHES.EORTC within a patient portal accompanied by educational and self-help material is possible as well.
Continuous assessment of patients' health status and QOL outside the hospital via telemonitoring can add to patient care. Especially patients undergoing active outpatient treatment might benefit from telemonitoring, as side effects and treatment-related symptom burden might occur outside the hospital . Alert systems enable health care professionals to recognise occurring symptoms in real time despite the geographic distance between the patient and medical services, contact patients and intervene appropriately . Research can also benefit from telemonitoring with EORTC measures as such data can be merged with routine assessments at the hospital and knowledge of the disease itself and treatment-related symptoms might be deepened.
Telemonitoring seems to be feasible for a considerable percentage of cancer patients  and  and patients show more and more interest in the use of electronic portals , which could provide, next to trustworthy educational and illustrating material, the possibility to complete EORTC measures as well. There is some evidence suggesting several benefits of using patient portals including the feeling of being better prepared for clinical appointments, development of better coping strategies, higher satisfaction with treatment choices, and better adherence to medical advice .
Nonetheless, efforts need to be made to include patients who are less likely to engage with electronic assessments due to unfamiliarity with or missing access to computer technology, lower interest in electronic health activities  and , or special needs (but auxiliary tools like adjustable font sizes, voice output or interactive voice response, additional input devices, telephone interviews may overcome this problem). Furthermore, implementation of telemonitoring needs to include a clear strategy on how to proceed if patient data indicates problems needing acute medical attention and how patients are effectively educated about the complementary nature of EORTC measures. Patients need to be aware that telemonitoring and using provided educational material and self-help advice must not replace or delay their visit to the hospital or doctor if they are feeling unwell.
6. Further areas of application of EORTC measures and ethical considerations
Routinely collected EORTC measures data can be a valuable source for comparing outcomes with predefined benchmarks, evaluation of practices, care pathways, policies, treatment technologies and disclosing systematic differences in distribution of health care among patients , , and . PRO data might even be used for certification requirements, as they stand for both the incorporation of patients' perceptions and patient empowerment.
Even though quality assurance can be a promising area for the use of PRO, there is a strong need for further research, as the interests of different stakeholders vary widely. So far, there are no standards on how PRO data have to be collected to consider the needs of all stakeholders without privileging or disadvantaging any of the parties involved, although some efforts have been undertaken to develop recommendations concerning this matter .
In the near future, EORTC measures can also be used for cost-utility analysis, and therefore for comparative effectiveness research . This has become possible with a recent project of the EORTC QLG and the international MAUCa (Multi-Attribute Utility in Cancer) group, within which the EORTC QLU-C10D measure was developed, a utility instrument enabling the estimation of quality adjusted life years using the EORTC QLQ-C30 data . Such analyses may focus on new treatment modalities or compare existing options, setting their monetary costs and QOL benefits and impairments against each other. Value sets have currently been developed for Australia, Germany, Poland, France and Italy and valuations in further European countries are planned.
Regarding ethical issues, it is important to balance the effort PRO assessments impose on patients with the expected benefit. Which and how many EORTC measures are used for data collection needs to be carefully chosen without underestimating the patients' ability and willingness to complete EORTC measures because they seem to be 'too ill'. The systematic exclusion of very ill patients alters data at a group level and may deprive exactly those patients who are in special need of particular interventions (e.g. referral to psychosocial services).
Before patients start completing EORTC measures, they need to be extensively informed about the purpose of assessments, issues concerning data confidentiality and security and their rights to withdraw from assessments and request destruction of data. Furthermore, disclosure needs to address common myths about electronic health interventions and telemonitoring (e.g. the fear that such interventions are offered because of disease progression, that the patient's own coping skills are insufficient , that there will be loss of personal contact with health care team).
Privacy is not only an issue if ePRO is conducted (adequate IT structure to ensure maximum data protection for collection, transmission and storage) but is also related to the environment of data assessment, as patients should be able to complete EORTC measures unobserved on their own in a private area, without any time constrains and the possibility to get help if necessary.
Protecting the confidentiality of patients' data might interfere with the usability of data in clinical routine. Some patients might not want all members of their clinical team to see their data, because they report sensitive information (e.g. sexual issues) or might fear negative consequences (e.g. following a report of low satisfaction with care). It is still a challenge to find adequate ways for data processing and access levels to preserve the interests of the parties involved.
This article briefly summarises the issues covered by our newly developed Manual for the use of EORTC measures in daily clinical practice, which aims at fostering the successful implementation of regular PRO assessments in routine patient care. The field of PRO assessments is continuously evolving and issues change regarding their importance and relevance. Medical, technical and regulatory advances and new scientific insights need to be taken into account when recommendations about the routine use of PROs are given. Consequently, guidelines like the presented EORTC manual need to be periodically updated. Possible topics to be included in a future version of the manual could be a more in-depth discussion on the use of EORTC measures for continuity of care, a critical appraisal of the possible use of the EORTC measures for quality assurance, a summary of the Mapi/EORTC QLG guidelines on the graphical presentation of EORTC measures in ePRO software, and the possible linkage between EORTC measures and self-management programs. As this manual is made for end-users of EORTC measures, any comments and suggestions for improvement and further development of the manual are warmly welcome: http://groups.eortc.be/qol/contact-us.
Conflict of interest statement
Bernhard Holzner is owner of the intellectual property rights of the CHES software. The other authors declare no conflicts of interest.
This work was supported by the EORTC Quality of Life Group [Lisa M. Wintner, grant number 00472012], the Austrian Science Fund (FWF) [Lisa M. Wintner, grant number P26930; Johannes M. Giesinger, grant number J3353]; and the Austrian National Bank (ÖNB) [Monika Sztankay, project number 14492).
Role of the funding source
All authors are active members of the EORTC Quality of Life Group. Neither the Austrian Science Fund nor the Austrian National Bank had any role in preparation, writing, proofs or submitting the manuscript whatsoever.
The work of Cheryl Whittaker, providing final language editing and proofreading, is highly appreciated.
-  J. Chen, L. Ou, S.J. Hollis. A systematic review of the impact of routine collection of patient reported outcome measures on patients, providers and health organisations in an oncologic setting. BMC Health Serv Res. 2013;13:211
-  J. Huebner, C. Rosé, J. Geissler, C.H. Gleiter, F.J. Prott, K. Muenstedt, et al. Integrating cancer patients' perspectives into treatment decisions and treatment evaluation using patient-reported outcomes–a concept paper. Eur J Cancer Care (Engl). 2014;23:173-179 Crossref
-  D. Miller, C. Steele Gray, K. Kuluski, C. Cott. Patient-centered care and patient-reported measures: let's look before we leap. Patient. 2015;8:293-299 Crossref
-  J.M. Valderas, A. Kotzeva, M. Espallargues, G. Guyatt, C.E. Ferrans, M.Y. Halyard, et al. The impact of measuring patient-reported outcomes in clinical practice: a systematic review of the literature. Qual Life Res. 2008;17:179-193 Crossref
-  D. Howell, S. Molloy, K. Wilkinson, E. Green, K. Orchard, K. Wang, et al. Patient-reported outcomes in routine cancer clinical practice: a scoping review of use, impact on health outcomes, and implementation factors. Ann Oncol. 2015;26:1846-1858
-  Association of Comprehensive Cancer Centres (ACCC) [Internet]. Oncoline cancer clinical practice guidelines. (, 2015) http://www.oncoline.nl/index.php [Accessed 2 December 2015]
-  German Cancer Society (Deutsche Krebsgesellschaft DI. Clinical practice guidelines. (, 2015) http://www.krebsgesellschaft.de/gcs/german-cancer-society/guidelines.html [Accessed 2 December 2015]
-  N.E. Adler, W.W. Stead. Patients in context—EHR capture of social and behavioral determinants of health. N Engl J Med. 2015;372:698-701 Crossref
-  M.B. Boyce, J.P. Browne, J. Greenhalgh. The experiences of professionals with using information from patient-reported outcome measures to improve the quality of healthcare: a systematic review of qualitative research. BMJ Qual Saf. 2014;23:508-518 Crossref
-  T. Locklear, B.J. Miriovsky, J.H. Willig, K. Staman, N. Bhavsar, K. Weinfurt, et al. Strategies for overcoming barriers to the implementation of patient-reported outcomes measures. An NIH health care systems research collaboratory patient reported outcomes core white paper (, 2014) https://sites.duke.edu/rethinkingclinicaltrials/tools-for-research/strategies-for-overcoming-barriers-to-the-implementation-of-patient-reported-outcomes-measures/ Available at:[Accessed 16 November 2014]
-  R.E. Jensen, N.E. Rothrock, E.M. DeWitt, B. Spiegel, C.A. Tucker, H.M. Crane, et al. The role of technical advances in the adoption and integration of patient-reported outcomes in clinical care. Med Care. 2015;53:153-159 Crossref
-  E. Basch, C. Snyder, K. McNiff, R. Brown, S. Maddux, M.L. Smith, et al. Patient-reported outcome performance measures in oncology. J Oncol Pract. 2014;10:209-211 Crossref
-  C.F. Snyder, S.M. Dy, D.E. Hendricks, J.R. Brahmer, M.A. Carducci, A.C. Wolff, et al. Asking the right questions: investigating needs assessments and health-related quality-of-life questionnaires for use in oncology clinical practice. Support Care Cancer. 2007;15:1075-1085 Crossref
-  C. Rodary, V. Pezet-Langevin, S. Garcia-Acosta, T. Lesimple, A. Lortholary, M.C. Kaminsky, et al. Patient preference for either the EORTC QLQ-C30 or the FACIT Quality of Life (QOL) measures: a study performed in patients suffering from carcinoma of an unknown primary site (CUP). Eur J Cancer. 2004;40:521-528 Crossref
-  International Society for Quality of Life Research (prepared by Aaronson N CA, Elliott T, Greenhalgh J, Halyard M, Hess R, Miller D, Reeve B, Santana M, Snyder C) [Internet]., User's guide to implementing patient-reported outcomes assessment in clinical practice. Available at: http://www.isoqol.org/UserFiles/file/UsersGuide.pdf. Version: November 11, 2011. [Accessed 24 June 2015].
-  C.F. Snyder, N.K. Aaronson, A.K. Choucair, T.E. Elliott, J. Greenhalgh, M.Y. Halyard, et al. Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations. Qual Life Res. 2012;21:1305-1314 Crossref
-  International Society for Quality of Life Research (prepared by Aaronson N CA, Elliott T, Greenhalgh J, Halyard M, Hess R, Miller D Reeve B, Santana M, Snyder C) [Internet]., User's guide to implementing patient-reported outcomes assessment in clinical practice. Available at: http://www.isoqol.org/UserFiles/2015UsersGuide-Version2.pdf. Version: January 2015. [Accessed 1 March 2016].
-  N.K. Aaronson, S. Ahmedzai, B. Bergman, M. Bullinger, A. Cull, N.J. Duez, et al. The European Organization for Research and Treatment of Cancer QLQ-C30: a quality-of-life instrument for use in international clinical trials in oncology. J Natl Cancer Inst. 1993;85:365-376 Crossref
-  M. Groenvold, M.A. Petersen, N.K. Aaronson, J.I. Arraras, J.M. Blazeby, A. Bottomley, et al. EORTC QLQ-C15-PAL: the new standard in the assessment of health-related quality of life in advanced cancer?. Palliat Med. 2006;20:59-61 Crossref
-  A. Brédart, A. Bottomley, J.M. Blazeby, T. Conroy, C. Coens, S. D'Haese, et al. An international prospective study of the EORTC cancer in-patient satisfaction with care measure (EORTC IN-PATSAT32). Eur J Cancer. 2005;41:2120-2131
-  EORTC Quality of Life Group [Internet]. Why do we need modules? http://groups.eortc.be/qol/why-do-we-need-modules. [Accessed 13 July 2015].
-  EORTC Quality of Life Group [Internet]. “Experimental” EORTC CAT http://groups.eortc.be/qol/experimental-eortc-cat. [Accessed 13 July 2015].
-  EORTC Quality of Life Group [Internet]. EORTC item bank http://groups.eortc.be/qol/item-bank.and.http://www.eortc.be/itembank2/Login.aspx. [Accessed 12 April 2016].
-  C. Johnson, N. Aaronson, J.M. Blazeby, A. Bottomley, P. Fayers, M. Koller, et al. Guidelines for developing quality of life questionnaires. (EORTC Publications, Brussels, 2011) http://groups.eortc.be/qol/sites/default/files/archives/guidelines_for_developing_questionnaire-_final.pdf [Accessed 26 May 2015]
-  N.W. Scott, P.M. Fayers, A. Bottomley, N.K. Aaronson, A. de Graeff, M. Groenvold, et al. Comparing translations of the EORTC QLQ-C30 using differential item functioning analyses. Qual Life Res. 2006;15:1103-1115 discussion 17–20 Crossref
-  N.W. Scott, P.M. Fayers, N.K. Aaronson, A. Bottomley, A. de Graeff, M. Groenvold, et al. A simulation study provided sample size guidance for differential item functioning (DIF) studies using short scales. J Clin Epidemiol. 2009;62:288-295 Crossref
-  N.W. Scott, P.M. Fayers, N.K. Aaronson, A. Bottomley, A. de Graeff, M. Groenvold, et al. The practical impact of differential item functioning analyses in a health-related quality of life instrument. Qual Life Res. 2009;18:1125-1130 Crossref
-  L. Dewolf, M. Koller, G. Velikova, C. Johnson, N. Scott, A. Bottomley, et al. EORTC quality of life group translation procedure. (, 2009) 32
-  S. Marshall, K. Haywood, R. Fitzpatrick. Impact of patient-reported outcome measures on routine practice: a structured review. J Eval Clin Pract. 2006;12:559-568 Crossref
-  G. Kotronoulas, N. Kearney, R. Maguire, A. Harrow, D. Di Domenico, S. Croy, et al. What is the value of the routine use of patient-reported outcome measures toward improvement of patient outcomes, processes of care, and health service outcomes in cancer care? A systematic review of controlled trials. J Clin Oncol. 2014;32:1480-1501 Crossref
-  E.E. Takeuchi, A. Keding, N. Awad, U. Hofmann, L.J. Campbell, P.J. Selby, et al. Impact of patient-reported outcomes in oncology: a longitudinal analysis of patient-physician communication. J Clin Oncol. 2011;29:2910-2917 Crossref
-  G. Velikova, A. Keding, C. Harley, K. Cocks, L. Booth, A.B. Smith, et al. Patients report improvements in continuity of care when quality of life assessments are used routinely in oncology practice: secondary outcomes of a randomised controlled trial. Eur J Cancer. 2010;46:2381-2388 Crossref
-  S.B. Detmar, M.J. Muller, J.H. Schornagel, L.D. Wever, N.K. Aaronson. Health-related quality-of-life assessments and patient-physician communication: a randomized controlled trial. JAMA. 2002;288:3027-3034 Crossref
-  P. Taenzer, B.D. Bultz, L.E. Carlson, M. Speca, T. DeGagne, K. Olson, et al. Impact of computerized quality of life screening on physician behaviour and patient satisfaction in lung cancer outpatients. Psychooncology. 2000;9:203-213 Crossref
-  D.L. Hilarius, P.H. Kloeg, C.M. Gundy, N.K. Aaronson. Use of health-related quality-of-life assessments in daily clinical oncology nursing practice: a community hospital-based intervention study. Cancer. 2008;113:628-637 Crossref
-  G. Velikova, L. Booth, A.B. Smith, P.M. Brown, P. Lynch, J.M. Brown, et al. Measuring quality of life in routine oncology practice improves communication and patient well-being: a randomized controlled trial. J Clin Oncol. 2004;22:714-724 Crossref
-  M.H. Frost, A.E. Bonomi, J.C. Cappelleri, H.J. Schünemann, T.J. Moynihan, N.K. Aaronson, et al. Applying quality-of-life data formally and systematically into clinical practice. Mayo Clin Proc. 2007;82:1214-1228 Crossref
-  S.B. Detmar, N.K. Aaronson. Quality of life assessment in daily clinical oncology practice: a feasibility study. Eur J Cancer. 1998;34:1181-1186 Crossref
-  S.A. McLachlan, A. Allenby, J. Matthews, A. Wirth, D. Kissane, M. Bishop, et al. Randomized trial of coordinated psychosocial interventions based on patient self-assessments versus standard care to improve the psychosocial functioning of patients with cancer. J Clin Oncol. 2001;19:4117-4125
-  K. Haywood, S. Marshall, R. Fitzpatrick. Patient participation in the consultation process: a structured review of intervention strategies. Patient Educ Couns. 2006;63:12-23 Crossref
-  L.I. Wagner, J. Schink, M. Bass, S. Patel, M.V. Diaz, N. Rothrock, et al. Bringing PROMIS to practice: brief and precise symptom screening in ambulatory cancer care. Cancer. 2015;121:927-934 Crossref
-  A.H. Krebber, F. Jansen, P. Cuijpers, C.R. Leemans, I.M. Verdonck-de Leeuw. Screening for psychological distress in follow-up care to identify head and neck cancer patients with untreated distress. Support Care Cancer. 2016;24:2541-2548
-  S. Duman-Lubberding, C.F. van Uden-Kraan, F. Jansen, B.I. Witte, L.A. van der Velden, M. Lacko, et al. Feasibility of an eHealth application “OncoKompas” to improve personalized survivorship cancer care. Support Care Cancer. 2016;24:2163-2171
-  S.K. Smith, K. Rowe, A.P. Abernethy. Use of an electronic patient-reported outcome measurement system to improve distress management in oncology. Palliat Support Care. 2014;12:69-73 Crossref
-  J.A. Talcott, J. Manola, R.C. Chen, J.A. Clark, I. Kaplan, A.V. D'Amico, et al. Using patient-reported outcomes to assess and improve prostate cancer brachytherapy. BJU Int. 2014;114:511-516 Crossref
-  Health and Social Care Information Centre [Internet]. Patient reported outcome measures http://www.hscic.gov.uk/proms. [Accessed 15 July 2015].
-  National Quality Forum [Internet]. Patient-reported outcomes in performance measurement. https://www.qualityforum.org/Publications/2012/12/Patient-Reported_Outcomes_in_Performance_Measurement.aspx: National Quality Forum; 2013. [Accessed 27 July 2015].
-  International Consortium for Health Outcomes Measurement (ICHOM) [Internet]. Lung cancer. (, 2015) http://www.ichom.org/medical-conditions/lung-cancer/ [Accessed 23 November 2015]
-  A.K. Morgans, A.C. van Bommel, C. Stowell, J.L. Abrahm, E. Basch, J.E. Bekelman, et al. Development of a standardized set of patient-centered outcomes for advanced prostate cancer: an international effort for a unified approach. Eur Urol. 2015;68:891-898
-  International Consortium for Health Outcomes Measurement (ICHOM) [Internet]. Advanced prostate cancer. (, 2016) http://www.ichom.org/medical-conditions/advanced-prostate-cancer/ [Accessed 2 February 2016]
-  International Society for Pharmacoeconomics and Outcomes Research ISPOR [Internet]. Quality of life promotional and label claim issues for drugs, biologicals, and devices. (, April 1999) http://www.ispor.org/workpaper/consensus/index.asp#issue1 Available at:[Accessed 20 July 2015]
-  J.M. Giesinger, L.M. Wintner, A. Zabernigg, E.M. Gamper, A.S. Oberguggenberger, M.J. Sztankay, et al. Assessing quality of life on the day of chemotherapy administration underestimates patients' true symptom burden. BMC Cancer. 2014;14:758 Crossref
-  M.C. Klee, M.T. King, D. Machin, H.H. Hansen. A clinical model for quality of life assessment in cancer patients receiving chemotherapy. Ann Oncol. 2000;11:23-30 Crossref
-  P.J. Hollen, R.J. Gralla, C.N. Rittenberg. Quality of life as a clinical trial endpoint: determining the appropriate interval for repeated assessments in patients with advanced lung cancer. Support Care Cancer. 2004;12:767-773 Crossref
-  P.A. Fayers, N. Bjordal, K. Groenvold, M. Curran, D. Bottomley. AslE-Usf-sp, font-family:“Verdana” s-s, mso-fareast-language:DE-AT, mso-bidi-language:AR-SA>. EORTC QLQ-C30 scoring manual. 3rd ed. (, 2001) Brussels
-  C.F. Snyder, A.L. Blackford, J.R. Brahmer, M.A. Carducci, R. Pili, V. Stearns, et al. Needs assessments can identify scores on HRQOL questionnaires that represent problems for patients: an illustration with the supportive care needs survey and the QLQ-C30. Qual Life Res. 2010;19:837-845 Crossref
-  C.F. Snyder, A.L. Blackford, N.K. Aaronson, S.B. Detmar, M.A. Carducci, M.D. Brundage, et al. Can patient-reported outcome measures identify cancer patients' most bothersome issues?. J Clin Oncol. 2011;29:1216-1220 Crossref
-  C.F. Snyder, A.L. Blackford, T. Okuyama, T. Akechi, H. Yamashita, T. Toyama, et al. Using the EORTC-QLQ-C30 in clinical practice for patient management: identifying scores requiring a clinician's attention. Qual Life Res. 2013;22:2685-2691 Crossref
-  M. Brundage, D. Feldman-Stewart, A. Leis, A. Bezjak, L. Degner, K. Velji, et al. Communicating quality of life information to cancer patients: a study of six presentation formats. J Clin Oncol. 2005;23:6949-6956 Crossref
-  M.D. Brundage, K.C. Smith, E.A. Little, E.T. Bantug, C.F. Snyder, Board PDPSA. Communicating patient-reported outcome scores using graphic formats: results from a mixed-methods evaluation. Qual Life Res. 2015;24:2457-2472
-  W. Kuijpers, J.M. Giesinger, A. Zabernigg, T. Young, E. Friend, I.M. Tomaszewska, et al. Patients' and health professionals' understanding of and preferences for graphical presentation styles for individual-level EORTC QLQ-C30 scores. Qual Life Res. 2016;25:595-604
-  D.M. Berwick. The science of improvement. JAMA. 2008;299:1182-1184 Crossref
-  P. Craig, P. Dieppe, S. Macintyre, S. Michie, I. Nazareth, M. Petticrew. Developing and evaluating complex interventions: the new Medical Research Council guidance. Int J Nurs Stud. 2013;50:587-592 Crossref
-  D.T. Eton, T.J. Beebe, P.T. Hagen, M.Y. Halyard, V.M. Montori, J.M. Naessens, et al. Harmonizing and consolidating the measurement of patient-reported information at health care institutions: a position statement of the Mayo Clinic. Patient Relat Outcome Meas. 2014;5:7-15 Crossref
-  C.F. Snyder, N.K. Aaronson. Use of patient-reported outcomes in clinical practice. Lancet. 2009;374:369-370 Crossref
-  J. Greenhalgh. The applications of PROs in clinical practice: what are they, do they work, and why?. Qual Life Res. 2009;18:115-123 Crossref
-  J. Greenhalgh, P. Abhyankar, S. McCluskey, E. Takeuchi, G. Velikova. How do doctors refer to patient-reported outcome measures (PROMS) in oncology consultations?. Qual Life Res. 2013;22:939-950 Crossref
-  Howell D, Bultz B, Fitch M, Groff S, Williams W, Cleghorn L. Ensuring a high-quality response to screening for distress data. Systematic knowledge translation is needed to improve patient experience. Available at: http://www.oncologyex.com/issue/2012/vol11-no3/online-feature_ensuring-a-high-quality-response.html. Oncology Exchange. 2012; 11:e35–e41.
-  A. Gilbert, D. Sebag-Montefiore, S. Davidson, G. Velikova. Use of patient-reported outcomes to measure symptoms and health related quality of life in the clinic. Gynecol Oncol. 2015;136:429-439 Crossref
-  G. Franx, M. Oud, J. de Lange, M. Wensing, R. Grol. Implementing a stepped-care approach in primary care: results of a qualitative study. Implement Sci. 2012;7:8 Crossref
-  R. Grol, J. Grimshaw. From best evidence to best practice: effective implementation of change in patients' care. Lancet. 2003;362:1225-1230 Crossref
-  J.M. Grimshaw, M.P. Eccles, J.N. Lavis, S.J. Hill, J.E. Squires. Knowledge translation of research findings. Implement Sci. 2012;7:50 Crossref
-  J. Lavis, H. Davies, A. Oxman, J.L. Denis, K. Golden-Biddle, E. Ferlie. Towards systematic reviews that inform health care management and policy-making. J Health Serv Res Policy. 2005;10(1):35-48 Crossref
-  L.A. Bero, R. Grilli, J.M. Grimshaw, E. Harvey, A.D. Oxman, M.A. Thomson. Closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. The Cochrane Effective Practice and Organization of Care Review Group. BMJ. 1998;317:465-468 Crossref
-  S. Carlfjord, M. Lindberg, P. Bendtsen, P. Nilsen, A. Andersson. Key factors influencing adoption of an innovation in primary health care: a qualitative study based on implementation theory. BMC Fam Pract. 2010;11:60 Crossref
-  M.J. Santana, L. Haverman, K. Absolom, E. Takeuchi, D. Feeny, M. Grootenhuis, et al. Training clinicians in how to use patient-reported outcome measures in routine clinical practice. Qual Life Res. 2015;24:1707-1718 Crossref
-  L. Warrington, K. Absolom, G. Velikova. Integrated care pathways for cancer survivors—a role for patient-reported outcome measures and health informatics. Acta Oncol. 2015;54:600-608 Crossref
-  K. Alsaleh. Routine administration of standardized questionnaires that assess aspects of patients' quality of life in medical oncology clinics: a systematic review. J Egypt Natl Canc Inst. 2013;25:63-70 Crossref
-  J. Hunter, K. Corcoran, S. Leeder, K. Phelps. Is it time to abandon paper? The use of emails and the Internet for health services research—a cost-effectiveness and qualitative study. J Eval Clin Pract. 2013;19:855-861
-  C.J. Gwaltney, A.L. Shields, S. Shiffman. Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: a meta-analytic review. Value Health. 2008;11:322-333 Crossref
-  Mapi Research Trust. Mapi Research Trust [Webpage], www.mapi-trust.org. [Accessed 9 April 2016].
-  R.E. Jensen, C.F. Snyder, A.P. Abernethy, E. Basch, A.L. Potosky, A.C. Roberts, et al. Review of electronic patient-reported outcomes systems used in cancer clinical care. J Oncol Pract. 2014;10:e215-e222 Crossref
-  A.V. Bennett, R.E. Jensen, E. Basch. Electronic patient-reported outcome systems in oncology clinical practice. CA Cancer J Clin. 2012;62:337-347
-  R. de Bree, I.M. Verdonck-de Leeuw, A.L. Keizer, A. Houffelaar, C.R. Leemans. Touch screen computer-assisted health-related quality of life and distress data collection in head and neck cancer patients. Clin Otolaryngol. 2008;33:138-142 Crossref
-  B. Holzner, J.M. Giesinger, J. Pinggera, S. Zugal, F. Schopf, A.S. Oberguggenberger, et al. The Computer-based Health Evaluation Software (CHES): a software for electronic patient-reported outcome monitoring. BMC Med Inf Decis Mak. 2012;12:126 Crossref
-  EORTC Quality of Life Group [Internet]. Electronic version CHES.EORTC http://groups.eortc.be/qol/electronic-version-cheseortc. [Accessed 14 July 2015].
-  R. Hawkins, S. Grunberg. Chemotherapy-induced nausea and vomiting: challenges and opportunities for improved patient outcomes. Clin J Oncol Nurs. 2009;13:54-64 Crossref
-  L. Balducci, G. Stanta. Cancer in the frail patient. A coming epidemic. Hematol Oncol Clin North Am. 2000;14:235-250 xi Crossref
-  H. Seybert, A. Lööf. Internet usage in 2010-households and individuals. Data Focus. 2010;50
-  N. Bush, G. Donaldson, C. Moinpour, M. Haberman, D. Milliken, V. Markle, et al. Development, feasibility and compliance of a web-based system for very frequent QOL and symptom home self-assessment after hematopoietic stem cell transplantation. Qual Life Res. 2005;14:77-93 Crossref
-  D.E. Gerber, A.L. Laccetti, B. Chen, J. Yan, J. Cai, S. Gates, et al. Predictors and intensity of online access to electronic medical records among patients with cancer. J Oncol Pract. 2014;10:e307-e312 Crossref
-  C.W. Arnold, M. McNamara, S. El-Saden, S. Chen, R.K. Taira, A.A. Bui. Imaging informatics for consumer health: towards a radiology patient portal. J Am Med Inf Assoc. 2013;20:1028-1036 Crossref
-  E. Kontos, K.D. Blake, W.Y. Chou, A. Prestin. Predictors of eHealth usage: insights on the digital divide from the Health Information National trends survey 2012. J Med Internet Res. 2014;16:e172 Crossref
-  J.Y. Wang, K. Bennett, J. Probst. Subdividing the digital divide: differences in internet access and use among rural residents with medical limitations. J Med Internet Res. 2011;13:e25 Crossref
-  S. Baffert, H.L. Hoang, A. Brédart, B. Asselain, S. Alran, H. Berseneff, et al. The patient-breast cancer care pathway: how could it be optimized?. BMC Cancer. 2015;15:394 Crossref
-  C.N. Wulff, P. Vedsted, J. Søndergaard. A randomised controlled trial of hospital-based case management to improve colorectal cancer patients' health-related quality of life and evaluations of care. BMJ Open. 2012;2
-  N. Black. Patient reported outcome measures could help transform healthcare. BMJ. 2013;346:f167 Crossref
-  A.W. Wu, H. Kharrazi, L.E. Boulware, C.F. Snyder. Measure once, cut twice—adding patient-reported outcome measures to the electronic health record for comparative effectiveness research. J Clin Epidemiol. 2013;66:S12-S20 Crossref
-  M.T. King, D.S. Costa, N.K. Aaronson, J.E. Brazier, D.F. Cella, P.M. Fayers, et al. QLU-C10D: a health state classification system for a multi-attribute utility measure based on the EORTC QLQ-C30. Qual Life Res. 2016;25:625-636
-  R. Norman, R. Viney, N.K. Aaronson, J.E. Brazier, D. Cella, D.S. Costa, et al. Using a discrete choice experiment to value the QLU-C10D: feasibility and sensitivity to presentation format. Qual Life Res. 2016;25:637-649
-  C. Sanders, A. Rogers, R. Bowen, P. Bower, S. Hirani, M. Cartwright, et al. Exploring barriers to participation and adoption of telehealth and telecare within the whole system demonstrator trial: a qualitative study. BMC Health Serv Res. 2012;12:220 Crossref
a Department of Psychiatry, Psychotherapy and Psychosomatics, Medical University of Innsbruck, Anichstr. 35, 6020, Innsbruck, Austria
b Division of Psychosocial Research and Epidemiology, The Netherlands Cancer Institute, Plesmanlaan 121, 1066, CX, Amsterdam, The Netherlands
c Quality of Life Department, EORTC Headquarters, Avenue E. Mounier 83, 1200, Brussels, Belgium
d The Research Unit, Department of Palliative Medicine, Bispebjerg Hospital, University of Copenhagen, Bispebjerg Bakke 23, 2400, Copenhagen, Denmark
e Netherlands Comprehensive Cancer Organisation, Gebouw Janssoenborch, Godebaldkwartier 419, 3511, DT, Utrecht, The Netherlands
f Leeds Institute of Cancer and Pathology, University of Leeds, LS2 9JT, Leeds, UK
g Clinical Psychology, VU University Medical Center, De Boelelaan 1118, 1081, HZ, Amsterdam, The Netherlands
∗ Corresponding author.
© 2016 Published by Elsevier B.V.