Response to the Forensic Science Regulator on Accreditation

Edit from our response to the Forensic Science Regulator

This is a shortened version of our response to the consultation paper from the Forensic Science Regulator in 2009 on Accreditation of Forensic Practitioners. The full response is available as a pdf file.

Purpose of the Defence Review

Accreditation

Council for the Registration of Forensic Practitioners

Discussion

Introduction and background

The Forensic Regulator has sought comment on his paper, “A Review of the Options for the Accreditation of Forensic Practitioners”. This response is made on behalf of The Forensic Institute.

The Forensic Institute

The Forensic Institute is a private firm which is normally instructed by the defence to assess the merit of the scientific evidence in criminal prosecutions.

Purpose of the Defence Review

The Forensic Institute maintains that the main purpose of a competent scientific defence expert is, rather than to simply repeat the Crown tests, to review the processes, procedures, and conclusions of the prosecution experts. This includes verification that what has been claimed to have been done has been done, and whether it has been done according to appropriate procedures and standards.

The main components of a thorough and competent defence review are verification and discovery.

Assessment by the Defence

Individual

Individual experts, with expertise in particular areas, will contribute to either the overall consideration of the case, or to specific reports on their area of expertise.

Collective

The Institute staff collectively discuss and peer review cases, case briefings and statements, and also reports received from other consultants, prior to sending these to the instructing lawyer. This is a form of quality assurance and may involve experts across the world.

Reports and briefings

The main output of all of the above is a collection of data, briefing papers, and reports that enable the assessment of the work performed to derive the opinion expressed in prosecution reports and, where appropriate, to form the basis for other expert reports which may challenge those opinions.

The final output is in the form of briefing papers and reports to instructing lawyers or Counsel. In some cases the briefings or reports may concur with the methods, interpretations and evaluations employed by the prosecution experts, in other cases there are differences between the prosecution and defence conclusions.

back to top

Accreditation

Accreditation is primarily a means by which methods and procedures are set to paper, and a system put in place to ensure that those methods are followed. The service specification for International Standards Organisation (ISO) accreditation is primarily set by ‘customer requirements’, not by scientific accuracy or reliability.

Accreditation and the “Customer”

Our role is to verify that the internal procedures have been performed by prosecution scientists, and that those procedures, where applicable, conform to nationally and internationally acceptable science. Although training and validation is examined as a small part of the accreditation process, there are no set standards, for accuracy or reliability of analyses for example, imposed upon the organisation.

The Scope of Accreditation

The ISO 17025 Standard does not specify any analytical standards of reliability or accuracy, merely that they are discovered via the validation process. It is this that enables a test to be described as ‘fit for purpose’ yet fail to achieve the necessary scientific acceptance to be deployed in scientific laboratories around the world.

Standards

Although National Occupational Standards have been developed for forensic practice they are, at best, very general guidelines. At the time when the Sector Skills Councils were being reorganised, it was decided that forensic science would fall under the remit of Skills for Justice and not the sector that it had until that point been under the auspices of; the Science, Engineering Manufacturing Technologies Alliance (SEMTA). In our view this diminished the scope for the wider scientific scrutiny of forensic science as distinct from some other forensic disciplines. This is of especial concern when it is being increasingly recognised that,

“… there is wide variability across forensic science disciplines with regard to techniques, methodologies, reliability, types and numbers of potential errors, research, general acceptability, and published material. Some of the forensic science disciplines are laboratory based (e.g., nuclear and mitochondrial DNA analysis, toxicology and drug analysis); others are based on expert interpretation of observed patterns (e.g., fingerprints, writing samples, toolmarks, bite marks, and specimens such as hair). The “forensic science community,” in turn, consists of a host of practitioners, including scientists (some with advanced degrees) in the fields of chemistry, biochemistry, biology, and medicine; laboratory technicians; crime scene investigators; and law enforcement officers. There are very important differences, however, between forensic laboratory work and crime scene investigations. There are also sharp distinctions between forensic practitioners who have been trained in chemistry, biochemistry, biology, and medicine (and who bring these disciplines to bear in their work) and technicians who lend support to forensic science enterprises.”

The same extensive and authoritative report states,

“disparities between and within the forensic science disciplines highlight a major problem in the forensic science community: The simple reality is that the interpretation of forensic evidence is not always based on scientific studies to determine its validity. This is a serious problem. Although research has been

done in some disciplines, there is a notable dearth of peer-reviewed, published studies establishing the scientific bases and validity of many forensic methods.”

It is therefore unlikely that an amalgam of ‘practitioners’ to decide on accreditation standards would have any more authority in some disciplines than a collection of astrologers or tea-leaf readers collectively deciding standards for themselves.

In addition to the analytical process,

“…in some cases, substantive information and testimony based on faulty forensic science analyses may have contributed to wrongful convictions of innocent people. This fact has demonstrated the potential danger of giving undue weight to evidence and testimony derived from imperfect testing and analysis. Moreover, imprecise or exaggerated expert testimony has sometimes contributed to the admission of erroneous or misleading evidence.”(ibid)

This refers to the evaluation and presentation of evidence which, currently, does not feature in ISO standards, nor is it immediately apparent how it can be.

back to top

The Council for the Registration of Forensic Practitioners

Although superficially attractive, we believe that the logic behind the creation of the CRFP was always fundamentally flawed.

Much, if not most, of the practice of forensic science is the practice of science, including the interpretation and evaluation of evidence with an occasional overlaying of legal considerations such as continuity. “Forensic science” is not some self-contained discipline like medicine, mycology, civil engineering, or computing.

The range of expertise required in Court is almost as wide as the entire human endeavour; from the individual with experience of specific items, systems, environments, or actions, to the individual with expertise and knowledge of these and more gained by systematic study.

Like the practice of science itself, in which there are no plain “scientists” because science is so specialised, there is no such thing as a “forensic scientist”. There are forensic pathologists, forensic biologists, forensic chemists, and many sub-disciplines. Rather than create a new body to simply embellish this professional knowledge with the particular skills required of the forensic component of a professional’s work, it would be better to supplement the remit of the professional body, in whatever discipline, with a forensic component.

If an expert has satisfied their own recognised profession of their competence, there seems no compelling reason why, having established their status as an expert, that they should have further requirements laid upon them when their expertise is sought by a court of law. If there is a perception that some specifically forensic skill is required, an extension of their ‘native’ professional body, which after all contains the core skills upon which they will be expected to rely, could fulfill this role.

back to top

Discussion

Although the Regulator is focussed on the provision of services to the Crown, there are specific references to the potential requirements on providers to the defence. We have set out herein the difference between the roles of these.

There is no evidence that we are aware of, however, that accredited organisations provide a more reliable and accurate service than non-accredited organisations. None of the high profile failures in forensic science of the past would necessarily have been prevented by accreditation. Indeed, all of the most recent errors have occurred within accredited systems. The evidence for any tangible improvement, and in particular any improvements in cost-efficiency or cost-effectiveness, is absent or scanty.

Not all providers are accredited. This has been recognised by the Forensic Science Regulator:

“Most forensic providers are accredited (all commercial providers involved in traditional laboratory based forensic work are accredited to ISO 17025 for their analytical work and certificated to ISO 9001 for their quality management systems).

Some police forces have quality management systems for their in-house forensic services and are certificated to ISO 9001. Some forces are moving towards accreditation to ISO 17025 for their in-house forensic laboratory functions. Other law enforcement bodies, such as the Serious Organised Crime Agency, have plans to move to a quality management system”.

In a similar principle to accreditation to ISO17025 or 9001, whilst accreditation has some meaning, nothing can be deduced on an organisation’s or individual’s competence from a lack of such accreditation. In particular, it is clear from our experience that most of the differences in opinion between scientists in Court arise from different evaluations of the data (i.e. what does it mean in the context of the case) rather than whether the physical analysis has been properly performed. Accreditation does not remove that area of dispute.

If it is intended that accreditation will be adduced as evidence of the correctness of the results and opinion delivered to Courts then it will be a necessary consequence that the defence will seek to scrutinise not only the production of those results but the accreditation process. To date the accreditation process has never been scrutinised thoroughly in Court despite the frequent claims of accredited organisations that their opinion is to be relied upon because of accreditation. However, when we made inquiry of UKAS during R v Hoey we were advised that the inspection process and details were “confidential”. If accreditation will be used to underpin the reliability of results, then the process used will need to be more open to inspection than it would appear to be at the moment.

First, for the reasons described above, accreditation in its present form, and even in ideal conditions, can never provide a guarantee for sound science. Science requires rigorous and on-going international validation. This can not be encompassed within accreditation schemes, which in order to be rigorous are necessarily cumbersome, time-consuming and expensive.

Second, even if accreditation provided a comprehensive and thoroughly reliable indication of how scientific evidence should be interpreted and evaluated, the fact remains that two scientists may have legitimately opposing views as to the significance of the evidence in a legal context. For this reason, accreditation can not be used as an alternative to the proper scrutiny of scientific evidence by the prosecution and defence.

Third, the Legal Services Commission may administer funds, but we would submit that it falls far short of any sort of expertise to determine whether or not forensic providers fulfill the requirements of a court expert (i.e. that they be experts and that they have special knowledge which may assist the court). Judges in the United Kingdom have been explicitly reluctant to take on the “gatekeeper” role of some US courts in determining whether or not expert witnesses (or entire areas of expertise) should be admitted to court. The practical effect of an approach in which the LSC decided on the expertise of any organisation or individual (rather than the Court or the instructed solicitor) is to transfer the gatekeeping function on expertise to the LSC; this is a highly undesirable outcome. It is our opinion that in some cases the LSC has come close to this error already.

Instead, the approach in the UK has been to allow those who can demonstrate expertise (through experience or qualifications) to give evidence, and for the weight of that evidence to be determined by the decision maker (the jury or judge) in the course of the trial. It is dangerous to suggest that the Legal Services Commission should circumvent this fundamental principle of the adversarial system and act as a pre-emptive authority on whom and what should be admitted to evidence.

In forensic science, a focus on “customer requirements”, the main parameter in ISO17025, is problematic because whilst both prosecution and defence experts are legally intended to be experts “for the court”, as a matter of practicality they are “instructed” by the Crown or the defence. Accreditation of forensic providers by the ISO is therefore fundamentally awkward because of the underlying philosophical tensions between the need for utility and value-for-money for the prosecution or defence, and the courts’ need for scientific reliability, validity and accuracy. Whilst the demands of the parties and the court are not necessarily mutually exclusive, the admissibility and credibility of expert witnesses should not be rubber-stamped by inadequate accreditation.

Our primary concern with the existing accreditation regime is that whilst it addresses the interpretation of results (i.e. how laboratories get actual results, for example, the amount of alcohol or drug in a sample or the alleles in a DNA profile), it does not adequately deal with the evaluation phase (i.e. the assessment of the evidential value of those results within the context of a specific case). The forensic evaluation of analytical results is by far the most common challenge to scientific evidence. The perspective of a prosecution scientist is necessarily informed by the material provided to them by police and the prosecution service. The defence perspective, in contrast, is informed by the fact that we are not obliged to present a particular view, but rather to test the view of the prosecution. Realistically, however, it is difficult to see how evaluation could ever be accredited, given that different experts can rationally hold different views on the evaluation of the same evidence. (It goes almost without saying that it would be no solution to set up an accreditation regime for the evaluation phase by allowing the “customer” to decide whether the evaluation is “fit for purpose”.)

Using accreditation as the basis for registration, and registration as the basis of expertise would not only exclude good scientists, but would also mask poor science, to the detriment of our criminal justice system.

Conclusion

Click here for the conclusion >>

back to top