BMJ 2002;324:1434-1437 ( 15 June )

Information in practice

Systematic review of cost effectiveness studies of telemedicine interventions

Pamela S Whitten, associate professor aFrances S Mair, senior lecturer bAlan Haycox, senior lecturer cCarl R May, professor of medical sociology dTracy L Williams, research fellow eSeth Hellmich, research assistant a

a Department of Telecommunications, Michigan State University, East Lansing, MI 48824-1212, USA, b Department of Primary Care, University of Liverpool, c Department of Pharmacology, University of Liverpool, d Centre for Health Services Research, University of Newcastle-upon-Tyne, e School of Primary Care, University of Manchester

Correspondence to: P S Whitten pwhitten{at}msu.edu


    Abstract
Top
Abstract
Introduction
Methods
Results
Discussion
References

Objectives: To systematically review cost benefit studies of telemedicine.
Design: Systematic review of English language, peer reviewed journal articles.
Data sources: Searches of Medline, Embase, ISI citation indexes, and database of Telemedicine Information Exchange.
Studies selected: 55 of 612 identified articles that presented actual cost benefit data.
Main outcome measures: Scientific quality of reports assessed by use of an established instrument for adjudicating on the quality of economic analyses.
Results: 557 articles without cost data categorised by topic. 55 articles with data initially categorised by cost variables employed in the study and conclusions. Only 24/55 (44%) studies met quality criteria justifying inclusion in a quality review. 20/24 (83%) restricted to simple cost comparisons. No study used cost utility analysis, the conventional means of establishing the "value for money" that a therapeutic intervention represents. Only 7/24 (29%) studies attempted to explore the level of utilisation that would be needed for telemedicine services to compare favourably with traditionally organised health care. None addressed this question in sufficient detail to adequately answer it. 15/24 (62.5%) of articles reviewed here provided no details of sensitivity analysis, a method all economic analyses should incorporate.
Conclusion: There is no good evidence that telemedicine is a cost effective means of delivering health care.

What is already known on this topic
The use of telemedicine has garnered much attention in the past decade

Hundreds of articles have been published claiming that telemedicine is cost effective

However, missing from the literature is a synthesis or meta-analysis of these publications

What this study adds
A comprehensive literature search of cost related articles on telemedicine identified more than 600 articles, but only 9% contained any cost benefit data

Only 4% of these articles met quality criteria justifying inclusion in a formalised quality review, and most of these were small scale, short term, pragmatic evaluations with few generalisable conclusions

There is little published evidence to confirm whether or not telemedicine is a cost effective alternative to standard healthcare delivery




    Introduction
Top
Abstract
Introduction
Methods
Results
Discussion
References

Proponents of telemedicine have championed its potential economic benefits in recent years. In Britain telemedicine systems have been proposed as a cost effective means of responding to structural problems in the organisation of the NHS. 1 2 In the United States telemedicine systems have been used, especially in rural areas, as a means of easing the problem of obtaining specialist advice and making referrals over wide distances.3 Even so, major questions about the cost effectiveness of telemedicine systems remain because the data used to support such claims are dispersed across studies of widely differing systems and diverse organisational contexts. Reported studies are often small in scale, methodologically flawed, and reflect pragmatic evaluations rather than controlled trials, making them unsuitable for formal meta-analysis.4 Given these problems, our objective was to address the question of whether the existing literature allows any conclusions to be drawn about the cost effectiveness of telemedicine interventions.

Telemedicine is defined here as clinical practice for diagnosis, review, or management undertaken synchronously or asynchronously through the medium of information and telecommunications technologies (excluding telephone and fax).


    Methods
Top
Abstract
Introduction
Methods
Results
Discussion
References

Two independent systematic reviews were started, one in the United States (Whitten and colleagues) and the other in Britain (Mair and colleagues). When each group became aware of the other's work it was decided to adopt a common strategy and share results, while still reviewing articles independently. As a result, both groups performed analyses on articles obtained through a common search strategy. Both groups categorised articles into those with or without data. The first group analysed articles without data by themes and articles with data by results. The second group determined which articles with data met quality criteria justifying inclusion in a quality review and performed that review.

Search strategy
We identified empirical studies of the costs associated with telemedicine interventions by searching the following electronic bibliographic databases: Medline (1966-June 2000), Embase (1988-June 2000), ISI Science Citation Index (1981-June 2000), ISI Social Science Citation Index (1981-June 2000), ISI Arts and Humanities Citation Index (1981-June 2000), and the database of the Telemedicine Information Exchange (TIE). Searching was restricted to articles in English language journals, with the keywords "telemedicine or telehealth or telemonitoring or telecommunications and cost or cost-effectiveness or economic or cost analysis or budget or financial or health care costs." We used multiple keyword sets to maximise results from the searches.

Data analysis
We divided the articles for analysis into two sets: those with and those without data. We categorised articles without data by theme, and achieved intercoder reliability of 90% for this thematic coding. For articles with data, we devised a data extraction instrument for coding aspects of each study. These included: location; type of service (such as specialty); delivery (synchronous or asynchronous); type of facility; duration and dates of study; number of patients, consultations, and images transmitted; cost revenue variables; and study conclusions.

The next stage required us to select articles with data for inclusion in a full review to assess their scientific quality. Variations in study design, patient groups, study context, and types of analysis meant that formal meta-analysis was inappropriate. For this reason, we present a qualitative analysis of the studies obtained. Although randomised controlled trials are the preferred source of evidence for inclusion in systematic reviews,5 hardly any such studies are to be found in the field of telemedicine. In view of this, we decided to include all published studies of telemedicine systems. When one project led to multiple reports (redundant publications), we reviewed the principal paper (with the greatest number of subjects) focusing on cost effectiveness. Our inclusion criterion for formal review was

  • Original research on telemedicine examining cost effectiveness of healthcare delivery.

Our exclusion criteria were

  • Papers reporting cost benefits of telemedicine used mainly for educational or administrative purposes
  • Papers reporting hypothetical cost analyses or modelling exercises without any associated formal clinical trial
  • Papers reporting economic analyses without any means of substantiating claimed resource use.

Two groups of reviewers examined the papers. Reviewers had expertise in telecommunications (PSW), family practice (FSM), health economics (AH) and sociology (CRM); all had experience with telemedicine services and research. The almost complete absence of randomised trials in the dataset meant that it was not possible to use traditional measures for assessing trial quality. However, two reviewers (FSM and AH) independently rated reports using an established checklist of criteria for assessing the quality of economic evaluations in health care (see box).6

Criteria used for assessing cost effectiveness studies6

  • Presence of a clear hypothesis
  • Presence or absence of a clear statement regarding the perspective from which cost effectiveness assessed
  • Methods
  • Presence or absence of a comparator and whether appropriate
  • Is the quality of medical evidence adequate?
  • Appropriate costs considered
  • Appropriate benefits considered
  • Assessment of whether a marginal analysis has been undertaken
  • Has a sensitivity analyses been undertaken?
  • Is the analysis appropriate to the local environment?




    Results
Top
Abstract
Introduction
Methods
Results
Discussion
References

Our literature search identified 612 articles, which we categorised as without cost data (n=557) or with cost data (n=55). We then thematically categorised the reports without cost data to determine the main premise for each article, and table 1 shows the primary categories.


                              
View this table:
[in this window]
[in a new window]
 

Table 1.  Primary categories of 557 articles examining cost effectiveness of telemedicine for healthcare delivery that lacked cost data

Quantitative analysis of reports with cost data
The 55 articles with data on costs and benefits reflected the diversity of the clinical and institutional contexts in which telemedicine is undertaken.w1-w55 Most of the reports (33, 60%) were from the United States; seven (13%) were from Norway, and the rest were from Australia, Canada, the rest of Europe, and Japan. Table 2 shows the specific variables used in the 55 studies, and table 3 lists the conclusions that were drawn, almost universally positive.


                              
View this table:
[in this window]
[in a new window]
 

Table 2.  Variables used in the cost analysis of the 55 articles examining cost effectiveness of telemedicine for healthcare delivery that had cost data


                              
View this table:
[in this window]
[in a new window]
 

Table 3.  Conclusions drawn in the 55 articles examining cost effectiveness of telemedicine for healthcare delivery that had cost data

Qualitative analysis of reports with cost data
Of the 55 studies, we subjected only 24 to full review with our checklist.w4-w6 w8 w13-w16 w19 w24 w26 w31 w33-w35 w39 w42 w48 w50-w55 (See extra table on bmj.com.) We excluded the other 31 studies because of duplication in publication of final results or their being limited to hypothetical or modelling exercises without formal trial data, being published only in abstract form, or providing insufficient information to assess the quality and reliability of the underlying study.

Although most of the 24 reports showed clear aims, only three constructed a formal hypothesis. Twenty were restricted to simple cost comparisons, rather than exploring wider implications for patient care. Only 11 studies specified the perspectives from which economic analysis was undertaken and so delineated the boundaries of analysis in a way that made their proper interpretation possible. Only four studies used a societal perspective, whereas the other 20 examined costs and benefits only from the narrow perspective of the healthcare provider (such as the hospital or prison). This partial approach impedes generalisability and neglects the comprehensive analysis of overall long term implications of a system---especially the analysis of opportunity costs, where health resources are valued in terms of the best alternative use for such resources.

Failure to make clear the perspective and boundaries of analysis was paralleled by use of inappropriate economic analytical techniques. Five of the studies used cost analysis, 15 used cost minimisation analysis, and four used cost effectiveness analysis. None of the studies used cost utility analysis, the conventional means of establishing the "value for money" that an intervention represents. Failure to calculate costs and benefits using an appropriate technique did not prevent considerable attention being paid to the potential economies of scale that would come about once telemedicine systems were more widely used. However, only seven studies attempted to explore the level of use that would be needed for telemedicine services to compare favourably with traditionally organised health care, and none addressed this question in sufficient detail to answer it adequately. Comparisons of costs with traditional forms of health care were absent in two studies and hypothetical in nine. (See extra table on bmj.com for details of the comparators and methods used in each study.)

Most of the studies entirely equated benefits with cost savings, with no analysis of changes in benefit to patients. Seventeen avoided the problem of benefits measurement by assuming equivalence with conventional medical practice on the basis of little or no efficacy data. This fundamental assumption of equivalence with conventional medical practice remains unproved.7

The short duration of projects (only 20% of interventions lasted longer than 12 months) introduced additional problems in interpreting the results. In any service evaluation, early point estimates of costs and benefits may not reflect costs and benefits that arise once the service has become established and is operating at a steady state. Sensitivity analysis is therefore essential to incorporate anticipated future changes in costs and benefits arising either through technological development, enhanced scale, or simply as a consequence of increasing familiarity with telemedicine. However, 15 of the studies simply relied on single point estimates obtained in a highly variable and dynamic environment.




    Discussion
Top
Abstract
Introduction
Methods
Results
Discussion
References

Our database search of English language articles identified only 55 that provided actual cost data on telemedicine interventions, and of these only 24 stood up to a full review using an established instrument for assessing the quality of economic evaluations. Most of the studies analysed were small scale, short term, pragmatic evaluations that added little to our knowledge of the costs and benefits that would be expected to result from the introduction of telemedicine services into routine clinical practice.

With such services it is difficult to generalise the results of individual cost effectiveness studies. In large part, comparative cost effectiveness of telemedicine systems depends on the unique local aspects of the individual service being evaluated. A telemedicine service that is cost effective in the remote highlands of Scotland is unlikely to generate the same cost effectiveness in the middle of Manchester. The difficulty of generating generalisable messages from evaluations undertaken in specific contexts emphasises the importance of assessing the local applicability of individual examples of the use of telemedicine. It is important to recognise that a service may be highly clinically and cost effective in one context but highly ineffective when transferred to another context in which accessibility and quality of local services are far higher.

Use of an established instrument to assess quality in economic evaluations serves to highlight the serious deficiencies in this body of peer reviewed and published literature. There was no uniformity of analysis, and the studies we reviewed were marked by poor design and inadequate technical quality. None of the studies used cost utility analysis to compare simultaneously variations in cost and outcomes. Cost utility analyses use outcome measures that are comparable and generalisable across different treatments as a means of comparing their "value for money." The lack of such studies means it is impossible to assess the extent to which telemedicine represents a sensible priority for healthcare investment. Until research in this area is conducted in accordance with generally accepted standards for economic analysis of health services, claims made on the basis of this body of literature should be regarded with considerable caution.

Conclusion
Reviews leading to negative conclusions are rarely welcomed. In the case of telemedicine, although claims about the utility and efficacy of new telecommunications systems in practice have been widely made,4 these are not founded on strong evidence. Given the paucity of methodologically sound studies producing robust and generalisable conclusions, there is presently no persuasive evidence about whether telemedicine represents a cost effective means of delivering health care.

The conventional response to such news is to emphasise the need for better designed and conducted trials, and these are certainly necessary. However, focusing on deficiencies in the design and conduct of research should not divert our attention from the other deficiencies revealed by this review. Peer review also seems to have failed to address adequately the range of basic errors in design and analysis that littered the studies we reviewed. Interested clinicians, policy makers, and healthcare providers should not assume that peer reviewed publication is necessarily an adequate guarantee of quality for economic evaluations of telemedicine.



    Acknowledgments

   Contributors: PSW is guarantor for the study and contributed to its conception and design, data analysis and interpretation, drafting the article, and final approval of the submitted paper. FSM contributed to the study conception and design, data analysis and interpretation, drafting the article, and final approval of the submitted paper. AH contributed to data analysis and interpretation and to drafting the article. CRM contributed to the study conception and design, data interpretation, and drafting the article. TLW contributed to data analysis and interpretation. SH contributed to the data analysis.

    Footnotes

Funding: None.

Competing interests: None declared.

Details of the studies reviewed are given in a table and reference list on bmj.com


    References
Top
Abstract
Introduction
Methods
Results
Discussion
References

1. Wootton R. Telemedicine in the National Health Service. J R Soc Med 1998; 91: 289-292.
2. NHS Executive. Information for health: an information strategy for the modern NHS 1998-2001. London: NHS Executive, 1998.
3. Peredinia DA, Allen AA. Telemedicine technology and clinical applications. JAMA 1995; 273: 483-488[CrossRef][ISI][Medline].
4. May C, Mort M, Mair FS, Ellis NT, Gask L. Evaluation of new technologies in health care: what's the context? Health Inform J 2000; 6: 64-68.
5. NHS Centre for Reviews and Dissemination. Report number 4. York: University of York, 1996.
6. Haycox A, Walley T. Pharmacoeconomics: evaluating the evaluators. Br J Clin Pharmacol 1997; 43: 451-456[Medline].
7. Currell R, Urquhart C, Wainwright P, Lewis R. Telemedicine versus face to face patient care: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2000;(2):CD002098.


© BMJ 2002

Related Articles

eHealth and the future: promise or peril?
Jeremy C Wyatt and Frank Sullivan
BMJ 2005 331: 1391-1393. [Extract] [Full Text] [PDF]

Email consultations in health care: Set your auto reply to "no"
Geoff Wong
BMJ 2004 329: 1046. [Extract] [Full Text]

Systematic review of cost effectiveness in telemedicine
Paul Scuffham, A R Haycox, and F S Mair
BMJ 2002 325: 598. [Extract] [Full Text]

Telemedicine is not cost effective
BMJ 2002 324: 0. [Full Text]

This article has been cited by other articles:

  • Murto, K., Bryson, G. L, Abushahwan, I., King, J., Moher, D., El-Emam, K., Splinter, W. (2008). Parents are reluctant to use technological means of communication in pediatric day care: [Les parents resistent a l'idee d'utiliser des moyens technologiques de communications dans les soins pediatriques de jour]. Canadian J. Anesthesia 55: 214-222 [Abstract] [Full text]  
  • Jackson, K. M., Scott, K. E., Graff Zivin, J., Bateman, D. A., Flynn, J. T., Keenan, J. D., Chiang, M. F. (2008). Cost-Utility Analysis of Telemedicine and Ophthalmoscopy for Retinopathy of Prematurity Management. Arch Ophthalmol 126: 493-499 [Abstract] [Full text]  
  • Bukachi, F., Pakenham-Walsh, N. (2007). Information Technology for Health in Developing Countries. Chest 132: 1624-1630 [Abstract] [Full text]  
  • Piette, J. D. (2007). Interactive Behavior Change Technology to Support Diabetes Self-Management: Where do we stand?. Diabetes Care 30: 2425-2432 [Full text]  
  • Hanson, E., Magnusson, L., Arvidsson, H., Claesson, A., Keady, J., Nolan, M. (2007). Working together with persons with early stage dementia and their family members to design a user-friendly technology-based support service. Dementia 6: 411-434 [Abstract]  
  • Brignell, M., Wootton, R., Gray, L. (2007). The application of telemedicine to geriatric medicine. Age Ageing 36: 369-374 [Abstract] [Full text]  
  • Shore, J. H., Brooks, E., Savin, D. M., Manson, S. M., Libby, A. M. (2007). An Economic Evaluation of Telehealth Data Collection With Rural Populations. Psychiatr. Serv. 58: 830-835 [Abstract] [Full text]  
  • British Thoracic Society Guideline Development Gro, (2007). Intermediate care--Hospital-at-Home in chronic obstructive pulmonary disease: British Thoracic Society guideline. Thorax 62: 200-210 [Full text]  
  • Daucourt, V., Sicotte, C., Pelletier-Fleury, N., Petitjean, M.-E., Chateil, J.-F., Michel, P. (2006). Cost-minimization analysis of a wide-area teleradiology network in a French region. Int J Qual Health Care 18: 287-293 [Abstract] [Full text]  
  • Wyatt, J. C, Sullivan, F. (2005). eHealth and the future: promise or peril?. BMJ 331: 1391-1393 [Full text]  
  • Kumar, S, Tay-Kearney, M-L, Constable, I J, Yogesan, K (2005). Internet based ophthalmology service: impact assessment. Br. J. Ophthalmol. 89: 1382-1383 [Full text]  
  • Wong, G. (2004). Email consultations in health care: Set your auto reply to "no". BMJ 329: 1046-1046 [Full text]  
  • Debnath, D (2004). Activity analysis of telemedicine in the UK. Postgrad. Med. J. 80: 335-338 [Abstract] [Full text]  
  • Finch, T., May, C., Mair, F., Mort, M., Gask, L. (2003). Integrating service development with evaluation in telehealthcare: an ethnographic study. BMJ 327: 1205-1209 [Abstract] [Full text]  
  • Bishai, D. M., Ferris, D. G., Litaker, M. S. (2003). What Is the Least Costly Strategy to Evaluate Cervical Abnormalities in Rural Women? Comparing Telemedicine, Local Practitioners, and Expert Physicians. Med Decis Making 23: 463-470 [Abstract]  
  • May, C., Harrison, R., Finch, T., MacFarlane, A., Mair, F., Wallace, P. (2003). Understanding the Normalization of Telemedicine Services through Qualitative Evaluation. J. Am. Med. Inform. Assoc. 10: 596-604 [Abstract] [Full text]  
  • Jacklin, P B, Roberts, J A, Wallace, P, Haines, A, Harrison, R, Barber, J A, Thompson, S G, Lewis, L, Currell, R, Parker, S, Wainwright, P (2003). Virtual outreach: economic evaluation of joint teleconsultations for patients referred by their general practitioner for a specialist opinion. BMJ 327: 84- [Abstract] [Full text]  
  • Hyler, S. E., Gangure, D. P. (2003). A Review of the Costs of Telepsychiatry. Psychiatr. Serv. 54: 976-980 [Abstract] [Full text]  
  • (2003). OTHER ARTICLES NOTED (Nov 01 to 18 Oct 02). Evid. Based Nurs. 6: e1-1 [Full text]  
  • Garceau, L., Henderson, J., Davis, L.J., Petrou, S., Henderson, L.R., McVeigh, E., Barlow, D.H., Davidson, L.L. (2002). Economic implications of assisted reproductive techniques: a systematic review. Hum Reprod 17: 3090-3109 [Abstract] [Full text]  
  • Scuffham, P., Haycox, A R, Mair, F S (2002). Systematic review of cost effectiveness in telemedicine. BMJ 325: 598-598 [Full text]  

Rapid Responses:

Read all Rapid Responses

Telemedicine too important to be derailed by flawed research
Albert J Kirshen
bmj.com, 15 Jun 2002 [Full text]
Misleading title in "This week in the BMJ"
Patricia A Macnair
bmj.com, 16 Jun 2002 [Full text]
Telemedicine field is ready for forthright analyses, including both positive and negative
Pamela Whitten
bmj.com, 18 Jun 2002 [Full text]
Cost-utility analysis is not always appropriate
Paul McCrone
bmj.com, 18 Jun 2002 [Full text]
Failure to compare cost effectiveness of civilian against military telemedicine is a red herring
Carl R May
bmj.com, 18 Jun 2002 [Full text]
Re: Economic evaluation is a broad church
Alan R Haycox
bmj.com, 20 Jun 2002 [Full text]
Telemedicine: Is cost effectiveness more important than clinical effectiveness and quality of life?
Giuseppe Riva
bmj.com, 20 Jun 2002 [Full text]
Half of cost effectiveness is cost
Robert C Hsiung
bmj.com, 21 Jun 2002 [Full text]
The Evidence Base Supporting The Clinical Effectiveness of Telemedicine Also Needs Strengthening
Dr Frances S Mair
bmj.com, 22 Jun 2002 [Full text]
Wasted times for telemedicine
James Sherifi
bmj.com, 26 Jun 2002 [Full text]
Quality of systematic reviews of cost-effectiveness studies
Paul A Scuffham
bmj.com, 27 Jun 2002 [Full text]
Lack of evidence for effectiveness differs from evidence of lack of effectiveness
Alan R Haycox, et al.
bmj.com, 4 Jul 2002 [Full text]



Student BMJ

Risk of surgery for inflammatory bowel disease: record linkage studies

What can you learn from this BMJ paper? Read Leanne Tite's Paper+

www.student.bmj.com

Listen to the latest BMJ Interview