The KHIT Blog: SPC for Lean Newbies

Sunday, June 9, 2013

SPC for Lean Newbies

I noted my concern a couple of posts back that the Lean Healthcare Transformation Summit 2013 appeared to be way light on the technical detail issue of SPC (Statistical Process Control) as a core component of the PDSA cycle that is otherwise touted as the foundation for the Lean process.

PDSA should really be "SPDSA" -- Study, Plan, Do, Study, Act.

I guess it's implicit in the "Plan" part (study the current state and incorporate the findings into your Plan). But, I didn't see much evidence of the quantification imperative of that in Orlando.

In fairness, my cautionary dubiety about "Six Sigma" aside, the DMAIC people are on point here (props to the Wiki):

Define
The purpose of this step is to clearly articulate the business problem, goal, potential resources, project scope and high-level project timeline. This information is typically captured within project charter document. Write down what you currently know. Seek to clarify facts, set objectives and form the project team. Define the following:

A problem statement

The customer(s)

Critical to Quality (CTQs) — what are the critical process outputs?

The target process subject to DMAIC and other related business processes

Project targets or goal

Project boundaries or scope

A project charter is often created and agreed upon during the Define step.

Measure
The purpose of this step is to objectively establish current baselines as the basis for improvement. This is a data collection step, the purpose of which is to establish process performance baselines. The performance metric baseline(s) from the Measure phase will be compared to the performance metric at the conclusion of the project to determine objectively whether significant improvement has been made. The team decides on what should be measured and how to measure it. It is usual for teams to invest a lot of effort into assessing the suitability of the proposed measurement systems. Good data is at the heart of the DMAIC process:

Identify the gap between current and required performance.

Collect data to create a process performance capability baseline for the project metric, that is, the process Y(s) (there may be more than one output).

Assess the measurement system (for example, a gauge study) for adequate accuracy and precision.

Establish a high level process flow baseline. Additional detail can be filled in later.

Analyze
The purpose of this step is to identify, validate and select root cause for elimination. A large number of potential root causes (process inputs, X) of the project problem are identified via root cause analysis (for example a fishbone diagram). The top 3-4 potential root causes are selected using multi-voting or other consensus tool for further validation. A data collection plan is created and data are collected to establish the relative contribution of each root causes to the project metric, Y. This process is repeated until "valid" root causes can be identified. Within Six Sigma, often complex analysis tools are used. However, it is acceptable to use basic tools if these are appropriate. Of the "validated" root causes, all or some can be

List and prioritize potential causes of the problem

Prioritize the root causes (key process inputs) to pursue in the Improve step

Identify how the process inputs (Xs) affect the process outputs (Ys). Data is analyzed to understand the magnitude of contribution of each root cause, X, to the project metric, Y. Statistical tests using p-values accompanied by Histograms, Pareto charts, and line plots are often used to do this.

Detailed process maps can be created to help pin-point where in the process the root causes reside, and what might be contributing to the occurrence.

Improve
The purpose of this step is to identify, test and implement a solution to the problem; in part or in whole. Identify creative solutions to eliminate the key root causes in order to fix and prevent process problems. Use brainstorming or techniques like Six Thinking Hats and Random Word. Some projects can utilize complex analysis tools like DOE (Design of Experiments), but try to focus on obvious solutions if these are apparent.

Create innovative solutions

Focus on the simplest and easiest solutions

Test solutions using Plan-Do-Study-Act (PDSA) cycle

Based on PDSA results, attempt to anticipate any avoidable risks associated with the "improvement" using FMEA

Create a detailed implementation plan

Deploy improvements

Control
The purpose of this step is to sustain the gains. Monitor the improvements to ensure continued and sustainable success. Create a control plan. Update documents, business process and training records as required.

A Control chart can be useful during the control stage to assess the stability of the improvements over time.

OK, thought experiment example. I Googled "control chart" and just picked one based on visual appeal.

So, let's call this Current State Customer Support Email Response Cycle Time and do a quick bit of Photoshopping. The idea here is cycle time from date/time receipt of a customer support email request to the time a response is recorded as "delivered" (not opened and read, just "delivered" -- because that's all we control).

I eyeballed and added the 2 sigma upper and lower "warning limit" lines in yellow.

Let's assume we culled a random sample of n=160 out of our support email server inbox. We find a current state of roughly two days response time, ~58 hours worst case. Sample appears to be roughly normally distributed (though we could test for that), and compliant with Gaussian assumptions for our purposes (though 2 CL "outliers" at n=160 begs a question; it's ~5x what we might expect by The Book. Still...).

Standard Deviation ("1 sigma") is 3.38 (I had to calculate this from the original data; no biggie).
C.V. ("Coefficient of Variation," a.k.a "Relative Standard Deviation" or "RSD") is ~7.1%, meaning we can unremarkably expect +/- 7.1% variation around the mean response time, current process (that's what "standard deviation" means -- expected variability).
The variation spread between the UCL and LCL, then, is about 42.5% relative to the mean.

The RSD is simply a measure of variability relative to the mean. It is useful. High RSD is a red flag, given that a core goal of any QI method is reduction of variation.

OK: Notwithstanding that this appears to be (in our thought experiment) a representative baseline random sample (no evident non-zero trendline, one basic marker of process instability), I'd be wanting to drill down deeper. But, that's another, more subtle issue.

For example, might we isolate all of the encounters which are, say, below -1 sigma (quicker response times), and look for any commonalities (i.e., identifiable "special causes")? As I noted in prior posts discussing HEDIS data examples, I might see a nominally random scatter depicting no apparent relation between cost and quality (below, CAD outcomes by cost proxies), but I'd be on the data-mining lookout for anything unique in that first quadrant. What are the people in the high-quality, low-cost segment doing right?

__

OK, so, back to our "control chart," we have some current state data. We then have to decide upon what will constitute a "significant" improvement should we undertake to try a process change. In science, you decide and document this prior to proceeding to your "Do" stage.

The salient (and difficult) question becomes one of declaring something along the lines of "we can reduce response cycle time by 20% with a concomitant reduction in variability" by doing "X".

At this point, "Do X," measure the upshot ("Study"), and "Act" on the basis of your findings.

This stuff is no "thought experiment" abstraction to me. It was my daily life in the 1980's in Oak Ridge (below). I painstakingly wrote the code that rendered this (PDF).

This is admittedly pedestrian"old school" QC stuff, but it's at the heart of being scientific.

ERRATUM

While attending the Lean Healthcare Transformation Summit 2013 "CEO Panel" discussion session last week, I had the irascible thought "my, my, -- what an incredibly diverse group of middle-aged white men." I noted the absence of women CEOs in a tweet.

This just came in my inbox.

THE BLOG COMMENT OF THE DAY, IN A NUTSHELL

Had to Photoshop it.

From a comment on The Health Care Blog today.
__

JUNE 11 MEANINGFUL USE STAGE 2 ITEM

...SNOMED CT clinical terminology is not widely adopted among providers and vendors, yet Stage 2 starts in October 2013 for hospitals. In particular, EHRs don’t capture communication codes present in 2014 CQMs, such as a specific code that conveys among physicians the degree of a medical condition, or “exclusion” codes that give a patient’s reason for declining medication or notes a patient doesn’t qualify for the medication, DeLano explains. Nor are most providers yet familiar with using SNOMED for clinical documentation, he adds.

Further, adopting SNOMED codes for clinical documentation is a major task, not so far from the complexity of ICD-10, DeLano contends, but the time needed to focus on SNOMED isn’t available as the industry adopts ICD-10. There are benefits to using SNOMED, but if providers and vendors aren’t ready for it, then they won’t be able to attest for meaningful use, he notes. “Providers think they are good because they are on a certified EHR product, but won’t get the clinical quality measures they want if the codes aren’t properly mapped.”

Asked if the federal government recognizes a gap in SNOMED readiness for Stage 2, Delano says, “I think there is awareness that there will be a shortfall in the reporting of CQMs.”...

Interesting. Concerns have been voiced over the utility of CQMs. e.g.,

Validity of electronic health record-derived quality measurement for performance monitoring

Amanda Parsons, Colleen McCullough, Jason Wang, and Sarah Shih
J Am Med Inform Assoc. 2012 Jul-Aug; 19(4): 604–609.
Published online 2012 January 16.

...We looked across the 11 clinical quality measures to assess where information was documented. The presence of data recognized for automated quality measurement varied widely, ranging from 10.7% to 99.9% (table 2). Measure components relying on vitals, vaccinations, and medications had the highest proportion of information documented in structured fields recognized by the automated quality measures. The majority of diagnoses for chronic conditions such as diabetes (>91.4% across measures), hypertension (89.3%), ischemic cardiovascular disease (>78.8% across measures) and dyslipidemia (75.1%) were documented in the problem list, a structured field used for automated quality measurement. Patient diagnoses not recognized for inclusion in the measure were recorded in the medical history, assessment, chief complaint, or history of present illness, sections that typically allow for free-text entries.

Diagnostic orders or results for mammogram had the lowest proportion (10.7%) of data recorded in structured fields recognized for automated quality measurement. The majority of the information for breast cancer screening was found as scanned patient documents and diagnostic imaging; both sources of information are not amenable for automated electronic queries.

Nearly half of the information for measures that require a laboratory test result, such as control of hemoglobin A1c and cholesterol, was documented in structured fields recognized for automated quality measurement (range 53.4–63.0%). Similarly, only half of the information regarding patient smoking status (53.4%) was recognized for automated quality measurement.

With the exception of medications, vaccinations, and blood pressure readings, practices varied substantially in where they chose to document the data elements required for automated quality measurement.

In estimating the denominator loss due to unrecognizable documentation, the average practice missed half of the eligible patients for three of the 11 quality measures—hemoglobin A1c control, cholesterol control, and smoking cessation intervention (table 3). No statistically significant differences were observed between the e-chart and EHR automated quality measurement scores in the number of patients captured for the denominator for the remaining eight measures. Current EHR reporting would underreport practice numerators for six of the 11 measures—hemoglobin A1c control, hemoglobin A1c screening, breast cancer screening, cholesterol control, cholesterol screening, and smoking status recorded.

...More studies are needed to assess the validity of EHR-derived quality measures and to ascertain which measures are best calculated using claims or administrative data or a combination of data sources. If provider-specific quality measurements are to be reported and made public, as is the plan for the meaningful use quality measures, further analysis is needed to understand the limitations of these data, particularly if they are prone to underestimation of true provider performance.

1 comment:

Mark GrabanJune 13, 2013 at 8:26 AM
I agree that an initial "study" is really part of the "plan" stage. Brian Joiner, in his book "Fourth Generation Management" was the first to write about the idea of "start with check," by the way. Great book.
ReplyDelete
Replies

Add comment

Brave New Health

Commonwell Health Alliance

Another important read (pdf)

I love this kind of stuff. It sustains and humbles me. "As politicians, advertisers, salesmen, and propagandists for various political, economic, moral, religious, psychic, environmental, dietary, and artistic doctrinaire positions know only too well, fallible human minds are easily tricked, by clever verbiage... Common language—or at least, the English language—has an almost universal tendency to disguise epistemological statements by putting them into a grammatical form which suggests to the unwary an ontological statement. A major source of error in current probability theory arises from an unthinking failure to perceive this."

Quotes

"An economist is a person who sees something that works in practice and tries to figure out whether it will work in theory."

- J.D. Kleinke, medical economist
___

"The only person who enjoys change is a baby with a wet diaper."

"Every misspent dollar in our health care system is part of somebody's paycheck.

- Brent James, M.D., M.Stat

“We could do healthcare, at markedly higher quality, for everyone in this country, without rationing or denying anybody the care that they need, without having the government dictate how doctors practice or whether hospitals could expand, at half the cost we do it now.”

- Health Care Futurist Joe Flower

Most of the sciences, unlike parts of medical science, are not concerned with the impossible. There is not complementary and alternative physics, or chemistry, or biochemistry, or engineering. These disciplines compare their ideas against reality, and, if the ideas are found wanting, abandoned."

- Mark A. Crislip, MD

"Q: How much alcohol is too much?
A: More than your doctor drinks."

- a physician I once heard speak during a CME presentation

“Just because science doesn’t know everything, doesn’t mean you get to fill in the gaps with whatever fairy tale most appeals to you.”

- Dara O’Briain

'[I]t is one small step from using the computer for "helping" doctors to monitoring them, judging them, dictating to them what to do, and withdrawing payment for computer non-compliance. The use of computer data is a multi-edged sword. It can be used for the "good," facilitating diagnosis and treatment and making it more accurate and up-to-date, and for “evil,” invading privacy, inviting security breechs, and making decisions based on the opinions of remote authorities rather than those present at the patient-doctor encounter.'

- Richard Reece, MD

“[T]here ARE statistics which are non-political. Just because The Washington Post/Fox News reports the temperature is 75 degrees doesn’t mean it’s really snowing and sunscreen is a liberal/conservative plot. Even if you earn a living being ideological.”

- Michael L. Millenson

"It is a generally a fairly convincing argument that people shouldn’t have to be subsidized to undertake a change which is in their best interest.

The reconciliation seems to be that EHR is not supposed to make a doctor’s practice more efficient and higher quality. It is supposed to make the system of care more efficient and higher quality, which is not the same thing. Those of you who took calc recall that maximizing the total of variables is not achieved by maximizing any one variable and this is a perfect example of that.

Those of you have served in combat certainly noticed that too — if everyone works as a team the unit takes fewer casualties. If you try to save your own hide, you might, but at the expense of more casualties overall."

- Al Lewis

"There are two ideas to keep in mind about Bayesian reasoning and how we tend to mess it up. The first is that base rates matter, even in the presence of evidence about the case at hand. This is often not intuitively obvious. The second is that intuitive impressions of the diagnosticity of evidence are often exaggerated."

- Daniel Kahneman, "Thinking, Fast and Slow"

"Physicians apply advanced scientific knowledge, but they must do so without the favorable conditions that experimental scientists create for themselves. Multitasking is forced on physicians, often in chaotic environments and under severe time and resource constraints."

- Lawrence and Lincoln Weed, "Medicine in Denial"

"It’s time to stop the whining about Obama care and acknowledge we already have universal health care. We just pay for it in the stupidest way possible that ensures problems are that much more disastrous and complicated when they’re finally treated."

- Mark Hoofnagle, MD, PhD

"Every act of conscious learning requires the willingness to suffer an injury to one's self-esteem. That is why young children, before they are aware of their own self-importance, learn so easily."

- Thomas Szasz, MD
___

"Of course, one reason that process metrics* are so popular is that processes are much easier to define and measure than outcomes."

- The Skeptical Scalpel
___

"There is an “illusion of validity” for any random data point, a seductive sense that is colored by what we hope will be true. Mountains of pharmaceutical claims are often made from mere molehills of data."

- Danielle Ofri, MD
___

"Joy empowers people. It is a source of energy that enables people to hope and plan and change their lives for the better. Spend some time around someone who is relentlessly negative and how do you feel–drained, right? More and more research shows that joy is not something that just happens to you, like a bolt of lightening out of the blue. Joy is, instead, a habit to cultivate. Negative thinking and despair are the crabgrass of our souls–weeds that take root and spread, sometimes to all areas of life. Joy, in contrast, is a soul’s rose–hardy when cared for, able to put down roots over time and withstand disease and extremes. Like a rose, however, your joy can become blighted from neglect or harsh conditions. We all need to tend to our joy–to prune away the badness, and to know that, even though it may look like a prickly bare root, if you invest time in a joyous outlook, gorgeous things will bloom, even in the harshest conditions."

- Dr. Jan Gurley
___

"'Solutions' exist only in mathematics."

- Karen Martin
___

"The issue of how to regulate clinical software is, in the long run, indistinguishable from the issue of how to regulate medicine. The only difference is that medicine is practiced in the open, without secrecy, subject to peer review, and under a merit-based state license."

- Adrian Gropper, MD
___

"Economist, rope, tree: some assembly required."

- Source unknown

DISCLAIMER:

I write this blog wholly on my own time and my own dime. The views proffered are expressly my own as a concerned and active citizen/taxpayer (in addition to being the result of my substantive experience in the various IT fields), and in no way reflect any policy views of my former employer, notwithstanding that some of the thinking has indeed obviously been spurred by the implications of the work with which I have been doing for them.

FAIR USE POLICY
I cite a ton of news and web sources spanning the breadth of relevant technical and policy domains, sometimes at substantial length. I believe I remain well within the bounds of "Fair Use," as [1] I am not doing any of this for profit, [2] I always provide attribution and links -- which, [3] far from negatively impacting any copyright holders' commercial interests, might actually increase traffic to and interest in their offerings.

Nonetheless, should I post anything of yours regarding which you have any objection, just let me know and I will remove it forthwith.

Search the KHIT Blog

Sunday, June 9, 2013

SPC for Lean Newbies

1 comment: