Jump to: Page Content, Site Navigation, Site Search,
You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.
Pamela S Whitten a Department of
Telecommunications, Michigan State University, East Lansing, MI
48824-1212, USA, b Department of Primary Care, University of Liverpool, c Department of
Pharmacology, University of Liverpool, d Centre for Health Services
Research, University of Newcastle-upon-Tyne, e School of Primary
Care, University of Manchester Correspondence to: P S
Whitten pwhitten{at}msu.edu
Abstract |
---|
Top
Abstract Introduction Methods Results Discussion References |
---|
Objectives:
To systematically review cost benefit
studies of telemedicine.
Design:
Systematic review of English language, peer reviewed journal articles.
Data sources:
Searches of Medline, Embase, ISI
citation indexes, and database of Telemedicine Information Exchange.
Studies selected:
55 of 612 identified articles that
presented actual cost benefit data.
Main outcome measures:
Scientific quality of reports
assessed by use of an established instrument for adjudicating on the
quality of economic analyses.
Results:
557 articles without cost data categorised by topic. 55 articles with data initially categorised by cost variables
employed in the study and conclusions. Only 24/55 (44%) studies met
quality criteria justifying inclusion in a quality review. 20/24 (83%)
restricted to simple cost comparisons. No study used cost utility
analysis, the conventional means of establishing the "value for
money" that a therapeutic intervention represents. Only 7/24 (29%)
studies attempted to explore the level of utilisation that would be
needed for telemedicine services to compare favourably with
traditionally organised health care. None addressed this question in
sufficient detail to adequately answer it. 15/24 (62.5%) of articles
reviewed here provided no details of sensitivity analysis, a method all
economic analyses should incorporate.
Conclusion:
There is no good evidence that
telemedicine is a cost effective means of delivering health care.
What is already known on this topic
Hundreds of articles have been published claiming that telemedicine is cost effective However, missing from the literature is a synthesis or meta-analysis of these publications What this study adds
Only 4% of these articles met quality criteria justifying inclusion in a formalised quality review, and most of these were small scale, short term, pragmatic evaluations with few generalisable conclusions There is little published evidence to confirm whether or not telemedicine is a cost effective alternative to standard healthcare delivery |
Introduction |
---|
Top
Abstract Introduction Methods Results Discussion References |
---|
Proponents of telemedicine have championed its potential economic benefits in recent years. In Britain telemedicine systems have been proposed as a cost effective means of responding to structural problems in the organisation of the NHS. 1 2 In the United States telemedicine systems have been used, especially in rural areas, as a means of easing the problem of obtaining specialist advice and making referrals over wide distances.3 Even so, major questions about the cost effectiveness of telemedicine systems remain because the data used to support such claims are dispersed across studies of widely differing systems and diverse organisational contexts. Reported studies are often small in scale, methodologically flawed, and reflect pragmatic evaluations rather than controlled trials, making them unsuitable for formal meta-analysis.4 Given these problems, our objective was to address the question of whether the existing literature allows any conclusions to be drawn about the cost effectiveness of telemedicine interventions.
Telemedicine is defined here as clinical practice for diagnosis,
review, or management undertaken synchronously or asynchronously through the medium of information and telecommunications technologies (excluding telephone and fax).
Methods |
---|
Top
Abstract Introduction Methods Results Discussion References |
---|
Two independent systematic reviews were started, one in the United States (Whitten and colleagues) and the other in Britain (Mair and colleagues). When each group became aware of the other's work it was decided to adopt a common strategy and share results, while still reviewing articles independently. As a result, both groups performed analyses on articles obtained through a common search strategy. Both groups categorised articles into those with or without data. The first group analysed articles without data by themes and articles with data by results. The second group determined which articles with data met quality criteria justifying inclusion in a quality review and performed that review.
Search strategy
We identified empirical studies of the costs associated with
telemedicine interventions by searching the following electronic
bibliographic databases: Medline (1966-June 2000), Embase (1988-June
2000), ISI Science Citation Index (1981-June 2000), ISI Social Science
Citation Index (1981-June 2000), ISI Arts and Humanities Citation Index
(1981-June 2000), and the database of the Telemedicine Information
Exchange (TIE). Searching was restricted to articles in English
language journals, with the keywords "telemedicine or
telehealth or telemonitoring or
telecommunications and cost or
cost-effectiveness or economic or cost
analysis or budget or financial or
health care costs." We used multiple keyword sets to maximise
results from the searches.
Data analysis
We divided the articles for analysis into two sets: those with and
those without data. We categorised articles without data by theme, and
achieved intercoder reliability of 90% for this thematic coding. For
articles with data, we devised a data extraction instrument for coding
aspects of each study. These included: location; type of service (such
as specialty); delivery (synchronous or asynchronous); type of
facility; duration and dates of study; number of patients,
consultations, and images transmitted; cost revenue variables; and
study conclusions.
The next stage required us to select articles with data for inclusion in a full review to assess their scientific quality. Variations in study design, patient groups, study context, and types of analysis meant that formal meta-analysis was inappropriate. For this reason, we present a qualitative analysis of the studies obtained. Although randomised controlled trials are the preferred source of evidence for inclusion in systematic reviews,5 hardly any such studies are to be found in the field of telemedicine. In view of this, we decided to include all published studies of telemedicine systems. When one project led to multiple reports (redundant publications), we reviewed the principal paper (with the greatest number of subjects) focusing on cost effectiveness. Our inclusion criterion for formal review was
Our exclusion criteria were
Two groups of reviewers examined the papers. Reviewers had expertise in
telecommunications (PSW), family practice (FSM), health economics (AH)
and sociology (CRM); all had experience with telemedicine services and
research. The almost complete absence of randomised trials in the
dataset meant that it was not possible to use traditional measures for
assessing trial quality. However, two reviewers (FSM and AH)
independently rated reports using an established checklist of criteria
for assessing the quality of economic evaluations in health care (see
box).6
Criteria used for assessing cost effectiveness
studies6
|
Results |
---|
Top
Abstract Introduction Methods Results Discussion References |
---|
Our literature search identified 612 articles, which we categorised as without cost data (n=557) or with cost data (n=55). We then thematically categorised the reports without cost data to determine the main premise for each article, and table 1 shows the primary categories.
|
Quantitative analysis of reports with cost data
The 55 articles with data on costs and benefits reflected the
diversity of the clinical and institutional contexts in which
telemedicine is undertaken.w1-w55 Most of the reports
(33, 60%) were from the United States; seven (13%) were from Norway,
and the rest were from Australia, Canada, the rest of Europe, and
Japan. Table 2 shows the specific variables used in the 55 studies, and table 3 lists the conclusions that were drawn, almost
universally positive.
|
|
Qualitative analysis of reports with cost data
Of the 55 studies, we subjected only 24 to full review with our
checklist.w4-w6 w8 w13-w16 w19 w24 w26 w31 w33-w35 w39 w42 w48
w50-w55 (See extra table on bmj.com.) We excluded the
other 31 studies because of duplication in publication of final results
or their being limited to hypothetical or modelling exercises without
formal trial data, being published only in abstract form, or providing insufficient information to assess the quality and reliability of the
underlying study.
Although most of the 24 reports showed clear aims, only three constructed a formal hypothesis. Twenty were restricted to simple cost comparisons, rather than exploring wider implications for patient care. Only 11 studies specified the perspectives from which economic analysis was undertaken and so delineated the boundaries of analysis in a way that made their proper interpretation possible. Only four studies used a societal perspective, whereas the other 20 examined costs and benefits only from the narrow perspective of the healthcare provider (such as the hospital or prison). This partial approach impedes generalisability and neglects the comprehensive analysis of overall long term implications of a systemespecially the analysis of opportunity costs, where health resources are valued in terms of the best alternative use for such resources.
Failure to make clear the perspective and boundaries of analysis was paralleled by use of inappropriate economic analytical techniques. Five of the studies used cost analysis, 15 used cost minimisation analysis, and four used cost effectiveness analysis. None of the studies used cost utility analysis, the conventional means of establishing the "value for money" that an intervention represents. Failure to calculate costs and benefits using an appropriate technique did not prevent considerable attention being paid to the potential economies of scale that would come about once telemedicine systems were more widely used. However, only seven studies attempted to explore the level of use that would be needed for telemedicine services to compare favourably with traditionally organised health care, and none addressed this question in sufficient detail to answer it adequately. Comparisons of costs with traditional forms of health care were absent in two studies and hypothetical in nine. (See extra table on bmj.com for details of the comparators and methods used in each study.)
Most of the studies entirely equated benefits with cost savings, with no analysis of changes in benefit to patients. Seventeen avoided the problem of benefits measurement by assuming equivalence with conventional medical practice on the basis of little or no efficacy data. This fundamental assumption of equivalence with conventional medical practice remains unproved.7
The short duration of projects (only 20% of interventions lasted longer than 12 months) introduced additional problems in interpreting the results. In any service evaluation, early point estimates of costs and benefits may not reflect costs and benefits that arise once the service has become established and is operating at a steady state. Sensitivity analysis is therefore essential to incorporate anticipated future changes in costs and benefits arising either through technological development, enhanced scale, or simply as a consequence of increasing familiarity with telemedicine. However, 15 of the studies simply relied on single point estimates obtained in a highly variable and dynamic environment.
Discussion |
---|
Top
Abstract Introduction Methods Results Discussion References |
---|
Our database search of English language articles identified only 55 that provided actual cost data on telemedicine interventions, and of these only 24 stood up to a full review using an established instrument for assessing the quality of economic evaluations. Most of the studies analysed were small scale, short term, pragmatic evaluations that added little to our knowledge of the costs and benefits that would be expected to result from the introduction of telemedicine services into routine clinical practice.
With such services it is difficult to generalise the results of individual cost effectiveness studies. In large part, comparative cost effectiveness of telemedicine systems depends on the unique local aspects of the individual service being evaluated. A telemedicine service that is cost effective in the remote highlands of Scotland is unlikely to generate the same cost effectiveness in the middle of Manchester. The difficulty of generating generalisable messages from evaluations undertaken in specific contexts emphasises the importance of assessing the local applicability of individual examples of the use of telemedicine. It is important to recognise that a service may be highly clinically and cost effective in one context but highly ineffective when transferred to another context in which accessibility and quality of local services are far higher.
Use of an established instrument to assess quality in economic evaluations serves to highlight the serious deficiencies in this body of peer reviewed and published literature. There was no uniformity of analysis, and the studies we reviewed were marked by poor design and inadequate technical quality. None of the studies used cost utility analysis to compare simultaneously variations in cost and outcomes. Cost utility analyses use outcome measures that are comparable and generalisable across different treatments as a means of comparing their "value for money." The lack of such studies means it is impossible to assess the extent to which telemedicine represents a sensible priority for healthcare investment. Until research in this area is conducted in accordance with generally accepted standards for economic analysis of health services, claims made on the basis of this body of literature should be regarded with considerable caution.
Conclusion
Reviews leading to negative conclusions are rarely welcomed.
In the case of telemedicine, although claims about the utility and
efficacy of new telecommunications systems in practice have been widely
made,4 these are not founded on strong evidence. Given the
paucity of methodologically sound studies producing robust and
generalisable conclusions, there is presently no persuasive evidence
about whether telemedicine represents a cost effective means of
delivering health care.
The conventional response to such news is to emphasise the need for better designed and conducted trials, and these are certainly necessary. However, focusing on deficiencies in the design and conduct of research should not divert our attention from the other deficiencies revealed by this review. Peer review also seems to have failed to address adequately the range of basic errors in design and analysis that littered the studies we reviewed. Interested clinicians, policy makers, and healthcare providers should not assume that peer reviewed publication is necessarily an adequate guarantee of quality for economic evaluations of telemedicine.
Acknowledgments |
---|
Contributors: PSW is guarantor for the study and contributed to its conception and design, data analysis and interpretation, drafting the article, and final approval of the submitted paper. FSM contributed to the study conception and design, data analysis and interpretation, drafting the article, and final approval of the submitted paper. AH contributed to data analysis and interpretation and to drafting the article. CRM contributed to the study conception and design, data interpretation, and drafting the article. TLW contributed to data analysis and interpretation. SH contributed to the data analysis.
Footnotes |
---|
Funding: None.
Competing interests: None declared.
Details of the studies reviewed are given in a table and reference list on bmj.com
References |
---|
Top
Abstract Introduction Methods Results Discussion References |
---|
1. | Wootton R. Telemedicine in the National Health Service. J R Soc Med 1998; 91: 289-292. |
2. | NHS Executive. Information for health: an information strategy for the modern NHS 1998-2001. London: NHS Executive, 1998. |
3. | Peredinia DA, Allen AA. Telemedicine technology and clinical applications. JAMA 1995; 273: 483-488[CrossRef][ISI][Medline]. |
4. | May C, Mort M, Mair FS, Ellis NT, Gask L. Evaluation of new technologies in health care: what's the context? Health Inform J 2000; 6: 64-68. |
5. | NHS Centre for Reviews and Dissemination. Report number 4. York: University of York, 1996. |
6. | Haycox A, Walley T. Pharmacoeconomics: evaluating the evaluators. Br J Clin Pharmacol 1997; 43: 451-456[Medline]. |
7. | Currell R, Urquhart C, Wainwright P, Lewis R. Telemedicine versus face to face patient care: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2000;(2):CD002098. |
Read all Rapid Responses
What can you learn from this BMJ paper? Read Leanne Tite's Paper+