Information about methods of integration for health care information technology





 

Jump to: Page Content, Site Navigation, Site Search,

You are seeing this message because your web browser does not support
basic web standards. Find out more about why this message is appearing
and what you can do to make your experience on this site better.
Search the BMJ

BMJ
BMJ Journals

BMJ Careers
BMJ Learning

Evidence Centre
BMJ Group

BMJ - helping doctors make better decisions Search this site:
Advanced search

Register for free services  Subscribe  Sign In
Home

Research
Methods and Reporting

Education
Clinical review

Practice
Shortcuts

Endgames
News

Comment
Editor's choice

Editorials
Letters

Rapid responses
Features

Observations
Head to head

Analysis
Views & reviews

Obituaries
Minerva

Fillers
Blogs

Podcast
Topics

Clinical topics
Non-clinical topics

Series
Theme issues

Video
Print Issues

Last seven days
Past weeks (Monday-Sunday)

Print issue
Rapid responses

Polls
Debates

Blogs
Podcasts

Video
Academic medicine

US highlights
BMJ USA 2001-2005

About BMJ
Explore the BMJ

Home > Research > BMJ 1998;316(7149):1959 (27 June), doi:
---------------------------------------------------------------------

BMJ 1998;316:1959-1961 ( 27 June )
Information in practice

Evaluating information technology in health care: barriers and
challenges
--------------------------------------------------------------
Heather Heathfield, senior lecturer, a David Pitty, director of
clinical IT, b Rudolph Hanka, director. c

a Department of Computing and Mathematics, Manchester Metropolitan
University, Manchester, b Healthcare Informatics Team, Royal Brompton
Hospital, London, c Medical Informatics Unit, Institute of Public
Health, University of Cambridge, Cambridge
Correspondence to: Dr Heathfield, Medical Informatics Unit, Institute
of Public Health, University of Cambridge, Cambridge CB2 2SR
H.Heathfield{at}doc.mmu.ac.uk

There is strong push for clinical leadership in the development and
procurement of information technology in health care.1 The lack of
clinical input to date has been cited as a major factor in the failure
of information technology in health services2 and has prompted many
clinicians to become involved in such endeavours. Furthermore, there
are various clinical decision support systems available, the merits of
which clinicians are expected to judge (such as Prodigy3 and Capsule4).
It is essential that clinicians have a knowledge of evaluation issues
in order that they can assess the strengths and weaknesses of
evaluation studies and thus interpret their results meaningfully, and
also contribute to the design and implementation of such studies to
provide them with useful information.

Summary points
---------------------------------------------------------------

Clinicians are becoming increasingly involved in the development
and procurement of information technology in health care, yet
evaluation studies have provided little useful information to
assist them
---------------------------------------------------------------

Evaluations by means of randomised controlled trials have not
yet provided any major indication of improved patient outcomes
or cost effectiveness, are difficult to generalise, and do not
provide the scope or detail necessary to inform decision making
---------------------------------------------------------------

Clinical information systems are a different kind of
intervention from drugs, and techniques used to evaluate drugs
(particularly randomised controlled trials) are not always
appropriate
---------------------------------------------------------------

The challenge for clinical informatics is to develop
multi-perspective evaluations that integrate quantitative and
qualitative methods
---------------------------------------------------------------

Evaluation is not just for accountability but to improve our
understanding of the role of information technology in health
care and our ability to deliver systems that offer a wide range
of clinical and economic benefits
The evaluation dilemma

Top
The evaluation dilemma
Problems with inappropriate eval
New directions in evaluation
Challenges for evaluating inform
Conclusions
References
Decision makers may be swayed by the general presumption that
technology is of benefit to health care and should be wholeheartedly
embraced. This view is supported by assertions such as that general
practitioner computing is seen "as an integral part of the NHS IT
strategy,"5 the US Institute of Medicine's statement that computing is
"an essential technology for healthcare,"6 and the increasingly high
levels of spending on healthcare information technology. On the other
hand, decision makers may support the argument that procurement of
information technology should be based on the demonstration, in
randomised controlled trials, of economic benefits or positive effects
on patient outcomes.7-12).

Regardless of which view you take, evidence is scarce. Large scale
pilot initiatives such as the NHS electronic patient record project
have yielded only anecdotal evidence, with little or no credence given
to results of external evaluation ("We now know how to do it and it is
achievable in the NHS"13). Results from economic analyses and
randomised controlled trials of healthcare systems are emerging, but
these studies cover only a small fraction of the total number of
healthcare applications developed and address a limited number of
questions, and most show no benefits to patient outcomes (D L Hunt et
al, Proceedings of the 5th Cochrane Colloquium, Amsterdam, October
1997).14
Those who base their judgment on the failure of randomised controlled
trials to show improved outcomes may cause important projects to be
prematurely abandoned and funding to be discontinued. In contrast,
those who heed the proponents of healthcare information technology and
base their decisions on unsubstantiated reports of projects, written
without external verification, may waste precious NHS resources
through the inappropriate and uninformed application of information
technology. This is likely to result in repeated failure without
retrospective insight, and so does nothing to further the science of
system development and deployment. The problem is confounded by the
fact that negative results are seen as unacceptable and do not
generally become public, thus failing to facilitate knowledge for
future developments.

Problems with inappropriate evaluations
Top
The evaluation dilemma
Problems with inappropriate eval
New directions in evaluation
Challenges for evaluating inform
Conclusions
References

Evaluation can be viewed as having a severe negative impact on the
progress of clinical information technology because, in our opinion,
many evaluation studies ask inappropriate questions, apply unsuitable
methods, and incorrectly interpret results. The evaluation questions
most often asked include those concerning economic benefits and
clinical outcomes, despite the lack of strong evidence of such and the
recognition of the difficulty of applying results in other contexts.15
The misplaced notion that clinical information technology is
comparable to a drug and should be evaluated as one has led to the
idea that the randomised controlled trial is the optimal method of
investigation.16 While a major deterrent to the use of randomised
controlled trials has been their cost, they are also vulnerable with
respect to external validity: trial results may not be relevant to the
full range of subjects (that is, specific implementations of a
healthcare application) or typical uses of a system in day to day
practice, and they are likely to cover only a small proportion of the
wide range of potential healthcare applications. Furthermore, negative
results from such trials cannot help us understand the effects of
clinical systems or build better ones in the future.
 
 View larger version (140K):
 in this window
 in a new window

New directions in evaluation
Top
The evaluation dilemma
Problems with inappropriate eval
New directions in evaluation
Challenges for evaluating inform
Conclusions
References

New perspectives on evaluation are emerging in the domain of health
care. Most important is the recognition that randomised controlled
trials cannot address all issues of evaluation and that a range of
approaches is desirable (Heathfield et al, Proceedings of HC96,
Harrogate, 1996).17 As pointed out by McManus, "Can we imagine how
randomised controlled trials would ensure the quality and safety of
modern air travel . . .? Whenever aeroplane manufacturers wanted to
change a design feature . . . they would make a new batch of planes,
half with the feature and half without, taking care not to let the
pilot know which features were present."18 Others have sought to find
surrogate process measures that may be used instead of "prohibitive"
outcome measures, thus making randomised controlled trials more cost
effective.19
Likewise, workers in clinical informatics have questioned the
usefulness of conducting randomised controlled trials on clinical
systems. The demonstration of quantifiable benefits in a randomised
controlled trial does not necessarily mean that end users will accept
a system into their working practices. Research shows that
satisfaction with information technology is more correlated with
users' perceptions about a system's effects on productivity than its
effect on quality of care.20-22

These insights have highlighted the need to examine professional and
organisational factors in system evaluation and have led to the
concept of multi-perspective, multi-method evaluations, which seek to
address a number of issues with multiple methods and with evaluators
from different backgrounds working together to produce an integrated
evaluation. This is coupled with an awareness of the importance of
qualitative methods in system evaluation.23-26 The NHS electronic
patient record project is an example of a large, multi-perspective
evaluation, which includes social scientists, health economists,
computer scientists, health service managers, and psychologists and
uses a wide range of different methods. However, the problems of
conducting large scale evaluations of this type show the need for
careful planning in such studies.27
Challenges for evaluating information technology in health care

Top
The evaluation dilemma
Problems with inappropriate eval
New directions in evaluation
Challenges for evaluating inform
Conclusions
References
Clinical systems are embedded social systems with different people,
institutions, providers, settings, and so on. While it is important
that we search for causal mechanisms that lead to clinical outcomes,
the investigation and, possibly, classification of such contexts is
essential. This will help us to understand and predict the behaviour
of systems and provide important knowledge to inform further
developments. This form of research will be facilitated by refocusing
attention from debates about specific methods towards issues of
multi-method evaluation and the integration of methods and results.

Conclusions
Top
The evaluation dilemma
Problems with inappropriate eval
New directions in evaluation
Challenges for evaluating inform
Conclusions
References

The arguments for performing multi-method evaluations must be
acknowledged and progressed within the community. Information
technology is not a drug and should not be evaluated as such. We
should look to the wider field of evaluation disciplines, in which
many of the issues now facing clinical informatics have been
addressed.
The current political context in which healthcare applications are
evaluated emphasises economic gains rather than quality of life. Thus,
the role of evaluation has been to justify past expenditures to
taxpayers, managers, etc, and so evaluation becomes a way of trying to
rebuild lost public trust. This is short sighted. Evaluation is not
just for accountability, but for development and knowledge building in
order to improve our understanding of the role of information
technology in health care and our ability to deliver high quality
systems that offer a wide range of clinical and economic benefits.

Acknowledgments
Funding: None.

Conflict of interest: None.
References

Top
The evaluation dilemma
Problems with inappropriate eval
New directions in evaluation
Challenges for evaluating inform
Conclusions
References
1.  Wyatt JC. Hospital information management: the need for clinical
  leadership. BMJ 1995; 311: 175-178Free Full Text.

2.  Heathfield HA, Wyatt J. Philosophies for the design and
  development of clinical decision-support systems. Methods Inf Med
  1993; 32(1): 1-8Medline.
3.  Wise J. Computer prescribing scheme gets green light. BMJ 1996;
  313: 250Free Full Text.

4.  Walton RT, Gierl C, Yudkin P, Mistry H, Vessey MP, Fox J.
  Evaluation of computer support for prescribing (CAPSULE) using
  simulated cases. BMJ 1997; 315: 37-38Free Full Text.
5.  Leaning M. The new information and management strategy of the
  NHS. BMJ 1993; 307: 217.

6.  In: Dick RS, Steen EB, eds. The computer-based patient record:
  an essential technology for health care. , Washington DC: National
  Academy Press, 1991.
7.  Lock C. What value do computers provide to NHS hospitals? BMJ
  1996; 312: 1407-1410Free Full Text.

8.  Donaldson LJ. From black bag to black box: will computers
  improve the NHS? BMJ 1996; 312: 1371-1372Free Full Text.
9.  Sullivan F, Elizabeth M. Has general practice computing made a
  difference to patient care? A systematic review of published
  reports. BMJ 1995; 311: 848-852Abstract/Free Full Text.

10.  Johnston ME, Langton KB, Haynes B, Mathieu A. Effects of
  computer-based clinical decision support systems on clinician
  performance and patient outcome. A critical appraisal of research.
  Ann Intern Med 1994; 120: 135-142Abstract/Free Full Text.
11.  Wyatt JC. Clinical data systems. Part 1: data and medical
  records. Lancet 1994; 344: 1543-1547Medline.

12.  Van der Loo JA. Overview of published assessment and evaluation
  studies. In: Van Gennip EMSJ, Talmon JL, eds. Assessment and
  evaluation of information technologies. , Amsterdam: IOS Press,
  1995:64-78.
13.  Brennan S, Dodds B. The electronic patient record programme: a
  voyage of discovery. Br J Healthcare Comput Inf Manage 1997; 14:
  16-18.

14.  Rotman BL, Sullivan AN, McDonald TW, Brown BW, DeSmedt P,
  Goddnature D, et al. A randomised controlled trial of a
  computer-based physician workstation in an outpatient setting:
  implementation barriers to overcome evaluation. J Am Med Inf Assoc
  1996; 3: 340-348Abstract/Free Full Text.
15.  Heathfield HA. Decision support systems. In: van Bemmel JH,
  McCray AT, eds. Yearbook of Medical Informatics 1995. , Stuttgart:
  Schattauer Verlagsgesellschaft mbH, 1995:455-457.

16.  Wyatt J, Spiegelhalter D. Evaluating medical expert systems:
  what to test and how? Med Inf 1990; 15: 205-217.
17.Mongerson PA. Patient's perspective of medical informatics. J
AmMed Inf Assoc 1995; 2: 79-84Abstract/Free Full Text.
18.  McManus C. Engineering quality in health care. Qual Health Care
  1996; 5: 127.

19.  Mant J, Hicks N. Detecting differences in quality of care: the
  sensitivity of measures of process and outcome in treating acute
  myocardial infarction. BMJ 1995; 311: 793-796Free Full Text.
20.  Lee F, Teich JM, Spurr CD, Bate DW. Implementation of physician
  order entry: user satisfaction and self-reported usage patterns. J
  Am Med Inf Assoc 1996; 3: 42-55Abstract/Free Full Text.

21.  Kaplan B. Information technology and three studies of clinical
  work. ACM SIGBIO Newsletter 1995; 15(2): 2-5.
22.  Igbaria M, Livari J, Maragahh H. Why do individuals use
  computer technology? A Finnish case study. Inf Manage 1995; 29:
  227-238.

23.  Friedman CP, Wyatt JC. Evaluation methods in medical
  informatics. New York: Springer-Verlag , 1997.
24.  In: Van Gennip EMSJ, Talmon JL, eds. Assessment and evaluation
  of information technologies in medicine. , Amsterdam: IOS Press,
  1995.

25.  Andersen JG, Aydin CE, Jay SJ. Evaluating healthcare
  information systems: methods and applications. Thousand Oaks, CA:
  Sage , 1994.
26.  Kaplan B. A model of a comprehensive evaluation plan for
  complex information systems: clinical imaging systems as an
  example. In: Brown A, Remeny D, eds. Proceedings of second
  European conference on information technology investment
  evaluation. , Birmingham: Operational Research Society,
  1995:174-181.

27. Heathfield HA, Hudson P, Kay S, Nicholson L, Peel V, Williams J,
  et al. Issues in multidisciplinary assessment of healthcare
  information systems. J IT People (in press).
(Accepted 28 April 1998)

---------------------------------------------------------------------
© BMJ 1998
Add to CiteULike CiteULike Add to Complore Complore Add to Connotea
Connotea Add to Del.icio.us Del.icio.us Add to Digg Digg
Add to Facebook Facebook Add to Reddit Reddit Add to StumbleUpon
StumbleUpon Add to Technorati Technorati Add to Twitter Twitter What's
this?

Relevant Article
----------------
Challenges to implementing the national programme for information
technology (NPfIT): a qualitative study
      Jane Hendy, Barnaby C Reeves, Naomi Fulop, Andrew Hutchings, and
      Cristina Masseria
      BMJ 2005 331: 331-336. Abstract Full Text PDF

This article has been cited by other articles:
----------------------------------------------
Scott, P. J., Briggs, J. S. (2009). A Pragmatist Argument for
Mixed Methodology in Medical Informatics. Journal of Mixed Methods
Research 3: 223-241 Abstract

Galligioni, E., Berloffa, F., Caffo, O., Tonazzolli, G.,
Ambrosini, G., Valduga, F., Eccher, C., Ferro, A., Forti, S.
(2009). Development and daily use of an electronic oncological
patient record for the total management of cancer patients: 7
years' experience. Ann Oncol 20: 349-352 Abstract Full text
Khong, S.-Y., Currie, I., Eccles, S. (2008). NHS Connecting for
Health and the National Programme for Information Technology. The
Obstetrician and Gynaecologist 10: 27-32 Abstract Full text

Peters, D. H., Kohli, M., Mascarenhas, M., Rao, K. (2006). Can
computers improve patient care by primary health care workers in
India?. Int J Qual Health Care 18: 437-445 Abstract Full text
Vreeman, D. J, Taggard, S. L, Rhine, M. D, Worrell, T. W (2006).
Evidence for Electronic Health Record Systems in Physical Therapy.
ptjournal 86: 434-446 Abstract Full text

Garg, A. X., Tonelli, M. (2005). The Tension Between E-Health
Innovation and E-Valuation. Arch Intern Med 165: 2329-2330 Full
text
Apkon, M., Mattera, J. A., Lin, Z., Herrin, J., Bradley, E. H.,
Carbone, M., Holmboe, E. S., Gross, C. P., Selter, J. G., Rich, A.
S., Krumholz, H. M. (2005). A Randomized Outpatient Trial of a
Decision-Support Information Technology Tool. Arch Intern Med 165:
2388-2394 Abstract Full text

Park, R. W., Shin, S. S., Choi, Y. I., Ahn, J. O., Hwang, S. C.
(2005). Computerized Physician Order Entry and Electronic Medical
Record Systems in Korean Teaching and General Hospitals: Results
of a 2004 Survey. J Am Med Inform Assoc 12: 642-647 Abstract
Full text
Adams, A., Blandford, A., Budd, D., Bailey, N. (2005).
Organizational communication and awareness: a novel solution for
health informatics. Health Informatics Journal 11: 163-178
Abstract

Fleet, L., Blandford, A. E. (2005). Requirements of time
management tools for outpatient physiotherapy practice. Health
Informatics Journal 11: 179-199 Abstract
Hendy, J., Reeves, B. C, Fulop, N., Hutchings, A., Masseria, C.
(2005). Challenges to implementing the national programme for
information technology (NPfIT): a qualitative study. BMJ 331:
331-336 Abstract Full text

Alberdi, E, Povyakalo, A A, Strigini, L, Ayton, P, Hartswood, M,
Procter, R, Slack, R (2005). Use of computer-aided detection (CAD)
tools in screening mammography: a multidisciplinary investigation.
Br. J. Radiol. 78: S31-S40 Abstract Full text
Delpierre, C., Cuzin, L., Fillaux, J., Alvarez, M., Massip, P.,
Lang, T. (2004). A systematic review of computer-based patient
record systems and quality of care: more randomized clinical
trials or a broader approach?. Int J Qual Health Care 16: 407-416
Abstract Full text

Finch, T., May, C., Mair, F., Mort, M., Gask, L. (2003).
Integrating service development with evaluation in telehealthcare:
an ethnographic study. BMJ 327: 1205-1209 Abstract Full text
Laerum, H., Karlsen, T. H, Faxvaag, A. (2003). Effects of
Scanning and Eliminating Paper-based Medical Records on Hospital
Physicians' Clinical Work Practice. J Am Med Inform Assoc 10:
588-595 Abstract Full text

van der Meijden, M J, Tange, H J, Troost, J, Hasman, A (2003).
Determinants of Success of Inpatient Clinical Information Systems:
A Literature Review. J Am Med Inform Assoc 10: 235-243 Abstract
Full text
Emery, J. D, Purves, I. N, Beaumont, R., Fahey, T., Mitchell, L.,
Sullivan, F., Montgomery, A., Gregor, P., van Wyk, J. T, Moorman,
P. W, van Wijk, M. A M, Eccles, M. P, McColl, E., Steen, N.,
Rousseau, N., Grimshaw, J., Parkin, D. (2003). Effect of
computerised evidence based guidelines. BMJ 326: 394-394 Full
text

Adams, A., Blandford, A. (2002). Acceptability of medical digital
libraries. Health Informatics Journal 8: 58-66 Abstract
Larum, H., Ellingsen, G., Faxvaag, A. (2001). Doctors' use of
electronic medical records systems in hospitals: cross sectional
survey. BMJ 323: 1344-1348 Abstract Full text

Atkinson, N. L., Gold, R. S. (2001). Online research to guide
knowledge management planning. Health Educ Res 16: 747-763
Abstract Full text
van der Kam, W., Meyboom de Jong, B, Tromp, T., Moorman, P., van
der Lei, J (2001). Effects of electronic communication between the
GP and the pharmacist. The quality of medication data on admission
and after discharge. Fam Pract 18: 605-609 Abstract Full text

Heathfield, H. A. (2001). Evaluating clinical systems: the social
programme perspective. Health Informatics Journal 7: 8-12
Abstract
Urquhart, C. J., Currell, R. A., Wainwright, P. J. (2000).
Evidence-based policy making in health informatics: indications
from systematic reviews of nursing record systems and
telemedicine. Health Informatics Journal 6: 204-211 Abstract

Delaney, B. C, Fitzmaurice, D. A, Riaz, A., Hobbs, F D R. (1999).
Can computerised decision support systems deliver improved quality
in primary care?. BMJ 319: 1281-1281 Full text
Emery, J., Walton, R., Coulson, A., Glasspool, D., Ziebland, S.,
Fox, J. (1999). Computer support for recording and interpreting
family histories of breast and ovarian cancer in primary care
(RAGs): qualitative evaluation with simulated patients. BMJ 319:
32-36 Abstract Full text

Heathfield, H., Louw, G. (1999). New challenges for clinical
informatics: knowledge management tools. Health Informatics
Journal 5: 67-73 Abstract
Rapid Responses:
----------------

Read all Rapid Responses
Untitled
      Bernard Fernando
      bmj.com, 7 Jul 1998 Full text

This Article
Respond to this article

Read responses to this article
Alert me when this article is cited

Alert me when responses are posted
Alert me when a correction is posted

View citation map
Services

Email this article to a friend
Find similar articles in BMJ

Find similar articles in PubMed
Add article to my folders

Download to citation manager
Request Permissions

Citing Articles
Read articles citing this article

Citing Articles via Google Scholar
Google Scholar

Articles by Heathfield, H.
Articles by Hanka, R.

Search for Related Content
PubMed

PubMed Citation
Articles by Heathfield, H.

Articles by Hanka, R.
Related Content

Clinical trials (epidemiology)
Information management

General practice / family medicine
Health economics

Health service research
Relevant Article

Find this article in its weekly table of contents
Bookmark with

Add to CiteULike Add to Complore Add to Connotea
 Add to Del.icio.us Add to Digg Add to Facebook Add to Reddit
 Add to StumbleUpon Add to Technorati Add to Twitter
 What's this?
---------------------------------------------------------------------

What's new
Last 7 days

Past weeks
Current print issue

Rapid responses
Blogs

Podcasts
Blogs

Podcasts
Keep updated

email
RSS Latest from BMJ

BMJ simple search
Follow BMJ on twitter

Watch on YouTube
Services

Submit an article
Subscribe/Activate

Request permissions
Tools Email to friend
Print this page Online pollTake Our poll(web poll) Find out more
Resources

Readers
Authors

Reviewers
Media

BMA members
Advertising and sponsors

Subscribers
Rapid responses for this article
--------------------------------

Untitled Bernard Fernando
More

Print issues
Current issue contents

Current cover image
Past issues

Subscribe

Specialist Libraries' Association top 100 journal

---------------------------------------------------------------------

---------------------------------------------------------------------

Access jobs at BMJ Careers Whats new online at Student BMJ

---------------------------------------------------------------------

Contact us - Privacy policy - Web site terms & conditions - Revenue
sources - Site map
HighWire Press - Feedback - Help - © 1998 BMJ Publishing Group Ltd.
i did should a best colorado health insurance provider online information an Like, up
visit - why but methods of integration for health care information technology as their your after
here best colorado health insurance provider online information be! hello munchies when once
methods of integration for health care information technology i he these can having about
themselves himself when has
ourselves this before have best colorado health insurance provider online information hello whom should be through is maybe know but
he why off up or her once after
did against both our methods of integration for health care information technology its while they myself
through each off their own had other doing
below same at why have those to ours yourselves or
once was know our our you best colorado health insurance provider online information be! yourselves
their about see because been own hello best colorado health insurance provider online information below some having health information privacy faq's yourself them me yours not
was yourselves further such itself surely should be ours below against our outta sight itself so under and myself
yourselves below before himself where and the
am or visit - your her own
once yourselves then down himself are be above yourself we these