Building Plants to Fill Up Your OSI PI System

by Oliver Yu

Speaking to a lot of prospects, I get this sense that the purpose of building and commissioning their plant is so the project engineer can finish their project. That the purpose of deploying control systems is to fill up their PI archives.

To me, that's wagging the dog.

You see, I'm the guy who uses the system. The cell culture engineer reporting applying statistical process control concepts to large-scale cell culture manufacturing. The guy who comes in after the professional the system integrator does the installation qualification (IQ) and operation qualification (OQ) and applies all that data to get a return for the company.

The system integrator is some automation engineer that works for a system integration company and his core competency is to deploy these systems. When that system integrator is done, he's off to another project at another company. He's not sticking around to actually use the system that he helped build.

Why is this important? A lot of prospects I meet are leaving the value on the table. They have uber-powerful systems like OSIsoft PI, but for the most part, don't use it. A lot of prospects think is there as a CYA-requirement... like something required to comply with cGMP.

The system integrator can tell you all about scan classes or data compression or instrument tags. The system integrator can't tell you about how best to build PI ProcessBook displays so your engineers can apply the data immediately... or the principles of PI Alias Nomenclature... or best practices for defining your equipment hierarchy.

Those items... crucial for getting a high ROI on your system... sit in this immense vacuous gap.

My opinion for what the solution is? A system integrator with real experience in your world, solving your problems with the tools you are trying to deploy.

For SCADA integrators, you're probably okay. For OSIsoft PI system integration? Go with a domain expert.

Biotech System Integrator for OSI PI

What are Biologics and How Are They Manufactured?

by Oliver Yu

Biologics are medicinal products created by a biological process - as opposed to chemical synthesis. What this means is that you can't get a couple beakers, Petri dishes and Bunsen burners in a lab and produce drugs like growth hormone.

Say you wanted growth hormone before Watson and Crick discovered DNA. You'd have to squeeze growth hormone out of the pituitary glands of pigs. This sucked because after you've gone through all the trouble of a sterile preparation, you're still left with porcine growth hormone. Worse, the manufacturing process is unscalable.

Now that we know about DNA, the manufacturing of biologics at large-scale is possible.

You see, medicine these days - as in drugs - are often complex proteins.

Proteins are molecules - composed of a sequence of amino acids - whose shape is determined by the sequence of said amino acids.

Amino acid sequence is determined by the DNA sequence.

So if you want to make biologics, you basically have to start with the protein... reverse engineer the amino acid sequence and then string together DNA that encodes for the amino acid sequence.

If you transfect (i.e. poke your DNA into) CHO cells (or other mammalian cells) and the DNA lands on a part of the chromosome that gets high ribosome traffic, then you have a cell line capable of producing the active pharmaceutical ingredient (API).

Biologics manufacturing then becomes growing the cells/having them secrete the API (a.k.a. cell culture) and subsequently purifying the drug.

Biologics made with recombinant DNA technology can be made reliably. Lean manufacturing principles can be applied and significant medical needs can be met.

Companies that own the infrastructure to identify marketable proteins and manufacture tons of it (while meeting regulatory requirements) are the ones with dominant positions.

More on cell culture.

Continuous Process Improvement and SPC

by Oliver Yu

Buzzwords are aplenty in this line of work: Lean manufacturing, lean six sigma, value stream mapping, business process management, Class A. But at the end of the day, we're talking about exactly one thing: continuous process improvement:

How to get your manufacturing processes (as well as your business process) to be better each day.

And to that, I say: "Pick your weapon and let's get to work." For me, I prefer statistical process control because SPC was invented in the days before continuous process improvement collided with information technology.

Back in those days, things had to be done by hand, concepts had to be boiled down in simple terms: special cause vs. common cause variability could simplify what was going on and clarify decision making. And having just Excel spreadsheets is a vast technological improvement to paper and pencil. In those days, there was no time for complexity of words and thought.

If we say words from the slower days of yesteryear, but use tools from today, we can solve a lot of problems and make a lot of good decisions.

Companies like Zymergi are third-party consultants who can help develop your in-house continuous process improvement strategy; especially for cell culture and fermentation companies. We focus applying statistical process control as well as knowledge management so that once we reduce process variability and increase reliability.

The technology is there to institutionalize the tribal knowledge so that when people leave, your high-paid consultants leave, the continuous process improvement know-how stays.

We use SPC and statistical analysis because it has been proven by others and it is proven by us. Data-driven decisions deliver real results.

7 Tools of SPC

1. Control Charts
2. Histograms
3. Correlations
4. Ishikawa Diagrams
5. Run Charts
6. Process Flow Diagrams
7. Pareto Charts

OSIsoft PI Scan Class and Exception Minimum

by Oliver Yu

Continuing on banalities of sub-second data, I got a really good question today about excmin vs. scan rate:

What is the relationship between scan rate and excmin?

Scan Rate

tells the interface how frequently to fetch the data

Excmin

tells the OSIsoft PI snapshot how frequently to ignore the data

It stands to reason that if you tell the interface to fetch sub-second data from the data source, then you don't want to throw out that data. If you do, you're simply clogging your network pipes with traffic that you end up tossing.

In practice, I have not seen excmin used. I'm sure it's useful when you absolutely, positively don't want data within excmin seconds of the previous point. But when you're worried about the 22 tag attributes for 5000 tags, that's a lot to worry about.

What typically happens is that PI tag data compression is handled with:

• Excmin=0
• Compmin=0
• Excdev>0
• Compdev> 0

Here's why:

We want to see what we're getting, so filtering out primary data just because it didn't fall inside of some arbitrary time-window is actually not GMP.

Secondly, filtering out primary data because it falls within the instrument accuracy of the measurement is a valid method for filtering out data because you are determining what instrument noise is and rationally filtering it out.

Get PI Compression Paper

So there it is:

• If you have sub-second data, set excmin and compmin = 0.
• Generally speaking, it's good practice to set both equal to 0 anyway
• Use excdev/compdev for data compression

OSIsoft PI OPC Interface and sub-second data

by Oliver Yu

I found out today that the PI-OPC Interface supports sub-second data. I'd imagine that this comes as no surprise to many of you, but it certainly does to me.

OSIsoft PI has supported the archival of sub-second data for quite some time. And for cell culture/fermentation processes, sub-second data is overkill. Cell culture happens over the course of days, production culture...weeks. Fermentations happen in hours, so very few things happen in between seconds.

Actually, there was this one time there was a rupture disk on a pasteurizing unit that had a setting of 50psi. When the transfer of media through the pasteurizer failed, due to the rupture disk, the highest pressure reading on OSIsoft PI was 29psi. As it turns, there was a pressure spike that happened on a sub-second basis, and was not captured... I suppose some large-scale manufacturing activities may require sub-second data for troubleshooting.

But ever since moving to sub-second data, it has been a pain because an event may happen at

18-Apr-12 01:24:03.5566

but if you were searching between

18-Apr-12 to 18-Apr-12 1:24, then you'd miss this event.

As is, people despise typing out more characters for specifying time. And in cell culture processes, it is simply not necessary. But as OSIsoft PI evolves to serve multiple industries, nuisances like sub-second data start cropping up.

In any case, the way to specify a sub-second scan-rate is at the interface. In the case of the PI-OPC Interface, you can specify the scan class as a fraction. If you wanted to specify scan rates at 1/10th, 1, 2, 5, 10, 30 and 60 seconds, your interface configuration file should read:

/f=0.1 /f=1 /f=2 /f=5 /f=10 /f=30 /f=60

Then any tag that you want to scan 10 times per second should have location4=1 (since 0.1 is the first scan class).

In any case,

• Few cell culture/fermentation processes require sub-second scan rates.
• Well duh: a PI system capable of archiving sub-second data has interfaces engineered to deliver data at sub-second scan rates.

Batch Relativity using OSI PI Batch

by Oliver Yu

It's the year 2012 and still, I see customers with batch processes not using PI Batch... the proven system for navigating batches in PI. Truth be told, some of these customers are not using OSI PI, which is in itself a problem.

Batch Relativity is having the the start/endtimes of a time-window so that when you need to look at a trend, you can plot it without having to manually input the timestamps:


Of course for time-windows in the recent past, you can use the arrows:


But for precise review of trends in the past, there are few alternatives to manual input.

When I was first starting out as a fermentation engineer, I distinctly recall getting the Gantt charts from the Planning & Scheduling department at the morning meetings and typing in estimates of the start/end times from the 11x17" paper I got each morning thinking there must be a better way. And there is: you can programmatically specify start/endtimes from PI Batch into the PI Trends.

If you have a batch manager, you can purchase software that writes to the PI Batch Database. For example, if you have an Emerson DeltaV system, you can purchase the Emerson DeltaV Batch Interface (EMDVB) that reads from EVT files and inserts records into the PI Batch database. Otherwise, you can use the native PI Batch Generator (PI BaGen) Interface that comes with the PI Server.

PI Batch Generator

To use the PI Batch Generator, there are several pre-requisites. The first is having an Active Tag.... a tag whose value = 1 when the batch is running and a tag whose value = 0 when the batch is stopped:


This is the minimum requirement for PI Batch Generator to work:

• A PI Unit for each unit you wish to track batches
• An Active Tag for each unit
• PI Batch Generator Interface installed as a service

For bioreactors (i.e. fermentors), if you don't have a tag that specifically starts/ends a batch, the tag you can use is the pH Controller Mode. Here's why:

You are generally interested in what goes on in the fermentor when there's something going on. And something is going on when it is batched with media. And when it is batched with media, pH control is typically ON; which means the pH Controller Mode = 1. On the back end of the batch, you typically turn of pH control after transfer or harvest so pH Controller Mode = 0 when the batch ends.

You'll know that you've picked the right point when your process values change when Active Tag = 1 and they flatline when Active Tag = 0:


For other types of process equipment, be clever with your existing tags to figure out the best Active Tag; for example, volume tends to be a good Active Tag.

With the Active Tag, you have satisfied the only requisite for using PI Batch Generator, all others are optional:

• Batch ID tag - a tag whose value equals the batch id at the time the batch is started.
  Typically some gibberish word that uniquely identifies this batch.
• Product tag - a tag whose value equals the name of the product being produced. (e.g. 'HER2', 'E25', 'VEGF')
• Procedure tag - a tag whose value equals manufacturing formula used

These values can be programmatically inserted in the event you don't want to consume tags for infrequent data.

Further reading:

• How PI Batch can increase your productivity at least 400%
• Intro to PI Batch Generator

Configure PI Batch!

Control Chart Limits vs. 3 StDev

by Oliver Yu

While control limits are approximations of 3 standard deviations, they are not 3 standard deviations.

In thermodynamics, we talk about state variables and path variables. State variables - like internal energy (U) … "is what it is." Other variables like work (w) are path variables… "its value depends on how you got there."

Standard deviation is a "state"-like parameter… if you have a set of points, the standard deviation is the standard deviation; it does not matter the order in which the data happened.



Using the same data from our previous control charting example, we see the standard deviation is 2.9 and a mean of 295. The 3 standard deviations around the average is 286 - 303.

Control limits, on the other hand, are path-like parameters that depend on the order in which it was received, and in the case of pretty random data, the control limits are 285 - 306... which is pretty close to the 3 standard devations, but not exact.




Viewing the control chart, it's obvious there are no special cause signals and there are no patterns in the data that indicate the data is out of the ordinary.

But suppose we got the same exact measurements... except this time, we found that each observed value was equal to or higher than the previous:




The standard deviation remains the same and therefore average +/- 3 standard deviations remains the same: 286 - 303. But look at the control limits... they have tightened significantly to 292 - 298.

This is because the control limits are computed from the moving range, and is when the same data shows an ascending pattern, the control limits are able to shrink and flag special cause signals where the standard deviations are not.

Apply 3 standard deviations where they are applicable; they are not applicable when identifying special cause signals of stable processes.

Control Charts for Bioprocesses

by Oliver Yu

A control chart is a graphical tool that helps you visualize process performance. Specifically, control charts help you visualize the expected variability of a process and unambiguously tells you what is normal (a.k.a. "common cause variability") and what is abnormal (a.k.a. "special cause variability").

Discerning common-cause from special-cause variability is crucial because responding to low results that are within expectation often induces more variability.

So up to this point, we know that low process variability allows us to detect changes to the process sooner. We also know that low process variability enables processes with higher capability.

Below is the control chart of the buffer osmo data from a previous blog post on reducing process variability.



The -green- horizontal line is the average of the population and the -red- lines are the control limits (upper control limit and lower control limit). Points that are within the UCL and LCL are expected (a.k.a. "common"). Points outside of the limits are unexpected (a.k.a. "special"). From the control chart, you can immediately see that the latest value of 301 mOsm/kg is "normal" or "common", and that no response is necessary.

Below, you see the control chart for the second set of data and how a reading of 297 mOsm/kg after 8 consecutive readings of 295 mOms/kg is anomalous and certainly worth an extra look.



There are all kinds of control charts and they have a rich history - worth reading if you're into that kind of thing. In batch/biologics processes, each data point corresponds with exactly one batch and so the type of control chart used is the IR chart.

It is important to know that the control limits are not computed from standard deviations - they are computed from the moving range... without going full nerd, the reason behind this is that control limits are sensitive to the order in which the points were observed and narrow when there is a trending pattern in the data.

Control charts for key process performance indicators are a must for any organization serious about reducing process variability. Firstly, control charts quantify variability. Secondly, control charts are easy to undertand. Lastly - and most importantly, control charts help marshall scarce resources by identifying common vs. special cause.

SPC - Control Charting

by Oliver Yu

No book on statistical process control is worth its salt if it fails to mention control charting; and true to the form of being solid on SPC, so does this pocket book:


Readers of this blog know well the necessity of control charting for process/campaign monitoring.

So it ought not to be surprising that we have yet another blog post about control charting. If you're really serious about reducing process variability, control charting is the highest impact, lowest cost method for establishing a baseline and understanding your status-quo process.

Everything that falls inside of the upper and lower control limits is expected variability (i.e. "common"). Since it is expected - don't do anything with it. Resist management tampering and don't waste resources investigating that which is expected.

Any point that falls outside of the upper and lower control limits is unexpected variability (i.e. "special"). Save your resources to investigate these points: chances are, you'll find something.

What hasn't been discussed here is within-control-limit patterns that can be considered special-cause. For example, 7-in-a-row on the same side of the centerline is a special cause even if no point has exceeded the control limit. Here are 4 other rules detailed later in the pocketbook:


And even farther on in the book are pages telling you how to compute control charts:


In this age with fast computers and JMP, it isn't a good use of your engineers' time to go back that far to derive the control charts limits.

Related posts:

• How to make IR control charts
• Control charts for Bioprocesses
• Control limits vs 3 standard deviations

SPC - Univariate and Bivariate Analysis

by Oliver Yu

The next tools in this SPC pocketbook are Histogram and Correlation.



In modern terms, these are called Univariate and Bivariate Analysis.

Histogram - aka Univariate Analysis

A histogram is one aspect of univariate analysis. According to the pocket book, the histogram is:

1. A picture of the distribution: How scattered are the data?
2. What the pattern of the data are (evenly-spread? Normal distribution?)
3. Can be used to compare the distribution to the specification

With modern computers, it is easy to create histograms with just a few clicks on your computer (with the $1,800 software JMP). In JMP, go to Analyze > Distribution.


You're going to get a dialog where you get to choose which columns you want to make into histograms. Select the columns and hit Y, Columns. Then click OK.


And voila, you get your histograms (plotted vertically by default) and more metrics than Ron Paul gets media coverage.


You get metrics like mean, standard deviation, standard error. And most importantly, you get visuals on how the data is spread.

Correlation - aka Bivariate Analysis

A correlation is also one specific type of bivariate analysis; the type where you plot numerical values against each other. Other types of bivariate analysis include means-comparisons and ANOVA. But yes, for SPC, the correlation is the most popular.

The pocketbook says that the correlation illustrates the relationship if it exists. From where I sit, the correlation feature is one of the most used functions in applying SPC to large-scale cell culture. Here's why:

While cell culture is complex, a lot of manufacturing phenomenon is simple. Mass-balance across a system is a linear process. Media batching is a linear process. The logarithm of cell density against time is a linear process. Many things can be explored by plotting Y vs. X and seeing if there's a correlation.

To get correlations with JMP, go to Analyze > Fit Y by X on the menu bar


You're going to get a dialog where you can specify which columns to plot on the y-axis (click Y, Columns). Then you get to specify which columns to plot on the x-axis (click X, Factor).


When you click OK, you're going to get your result. If it turns out that your Y has nothing to do with X, you're going to get something like this: a scatter of points where the mean and the correlation basically are on top of each other.


If you get a response that does vary with the factor, you're going to get something like this:



SPC in the information age is effortless. There really is no excuse to not have data-driven decisions that yield high-impact results.

SPC - Cause/Effect Diagrams and Run Charts

by Oliver Yu

The next two tools were used constantly for large-scale manufacturing sciences support of cell culture: Cause Effect Diagram and the Run Chart.

Cause/Effect (Ishikawa) Diagram

The cause/effect diagram (aka) Ishikawa diagram is essentially taxonomy for failure modes. You break down failures (effects) into 4 categories:

1. Man
2. Machine
3. Method
4. Materials

It's used as a brainstorming tool to put it all out there and to help visualize how an event can cause the effect. This was particularly helpful contamination investigations. In fact, there's a "politically correct" Ishikawa diagram in my FREE case study on large-scale bioreactor contamination.

Get Contamination Cause/Effect Diagram

The cause/effect diagram helps clarify thinking and keeps the team on-task.

Run Chart

The Run Chart is basically what a chart-recorder spits out. In this day and age, it's what we call OSIsoft PI. You plot a parameter against time (called a trend), and when you do this, you get to see what's happening in sequential order. When you plot a lot of parameters on top of one another, you begin to understand sequence. Things that happen later cannot cause events that happened earlier. Say your online dissolved oxygen readings spiked below 5% for 10 seconds, yet your pO2 remains steady and the following viability measurement shows no drop off in cell viability, you can basically say that the dO2 spike was measurement error.

Here's an example of the modern-day run chart, it's called, "PI":


Run charts (i.e. PI) are crucial for solving immediate problems. A drifting pH probe can dump excess CO2 into a media-batched fermentor. Being able to see real-time data from your instruments and have the experience to figure out what is going on is key to troubleshooting large-scale cell culture and fixing the problem real-time so that the defect is not sent downstream.

Get #1 Biotech/Pharma PI Systems Integrator

As you can see, SPC concepts are timelessly applied today to cell culture and fermentation... albeit with new technology.

SPC - Process Flow Diagram/Pareto Charts

by Oliver Yu

So that little SPC Book goes into 7-tools to use, the next page goes into Process Flow Diagrams and Pareto charts.

Process Flow Diagram

The first tool appears to be the Process Flow Diagram[tm], where one is supposed to draw out the inputs and outputs of each process step. I suppose in the "Lean" world, this is the equivalent of value-stream mapping.

The text of the booklet calls it a

Pictoral display of the movement through a process. It simply shows the various process stages in sequential order.

Normally, I see this on a Powerpoint slide somewhere. And frankly, I've rarely seen it used in practice. More often, if we show this to consultants to get them up to speed.

Pareto Chart

The pareto chart is essentially a pie chart in bar-format. The key difference is that pie charts are for the USA Today readership while pareto charts are for real engineers -- this is to say that if you're putting pie charts in Powerpoint and you're an engineer, you're doing it wrong.

Pareto charts are super useful because they help figure out your most pressing issue. For example, say you're create a table of your fermentation failures:


So you have counted the number of observed failures alongside a weight of how devastating the failure is. Well, in JMP, you can simply create a pareto chart:


and out pops a pareto chart.


What this pareto chart shows you is the most important things to focus your efforts on. If you solve the top 2 items on this pareto chart, you will have solved 80% of your problems - on a weighted scale.

The pareto is a great tool for metering out extremely limited resources and has been proven extremely effective in commercial cell culture/fermentation applications.

SPC - Deming 14 Points for Management

by Oliver Yu

I was cleaning out my bookshelf and found this nifty little pocketbook.


Quality policies back then were not run-on paragraphs:

Dow Corning will provide products and services that meet the requirements of our customers. Each employee must be committed to the goal of doing it right the first time.

Page 4 contains Deming's 14 points for management; apparently, Deming didn't know that humans can remember in groupings of items in 3, 5, or 7:

1. Create constancy of purpose toward improvement of product and service.
2. Adopt the new philosophy. Acceptance of poor product and service is roadblock to productivity.
3. Cease dependence on mass inspection. Replace by improved processes.
4. End the practice of awarding business on basis of price tag alone.
5. Find problems and fix them. Continually reduce waste and improve quality.
6. Institute modern methods of on training on the job.
7. Institute modern methods of supervision.
8. Drive out fear.
9. Break down barriers between departments and locations.
10. Eliminate numerical goals, posters and slogans. Don't ask for new levels of productivity without providing new methods.
11. Eliminate work standards and numerical quotas.
12. Remove barriers that stand between the worker and his right to pride of worksmanship.
13. Institute a vigorous program of education and training.
14. Create a structure in top management that will push every day on the above 13 points.

Page 5 is an introduction to the seven tools described in the remaining 25 pages of this pocket book.

You Suck at Reducing Cell Culture Contaminations

You do, you're awful. But that's why you're here.

Reducing cell culture contaminations is a big deal.

Not big but HUGE.

It's huge simply because contemporary contamination rates are typically between five to 15% and can skyrocket at any time.

You know what I'm talking about. You're nervously awaiting the next bioreactor contamination right now.

Chances are that you've sat through a contamination investigation meeting NOT getting to the bottom of why your cell cultures are coming down contaminated.

You're going through the motions of a contamination response procedure that's in an SOP somewhere because it is GMP to have written procedures and this cross-functional team of warm-bodies isn't pulling you out of this tailspin.

I mean, even if it wasn't for having to sit through crappy meetings, we're still talking about a biological process being out-of-control. Being out-of-control means you are not GMP.

Google Genzyme and you'll see their contamination problems and the subsequent FDA beatdown.

Their Allston plant at 500 Soldiers Field Rd (right next to the Harvard Bschool) faced rampant viral contamination of their bioreactors. These contaminations temporarily shuttered their plant, left patients facing drug shortages, brought in a $175 million consent decree and crashed the stock price.

On a personal note, I interviewed for a position at the Allston plant in 1998 and was rejected; in hindsight, that rejection was one of the better things that happened for my career.

The idea is simple: stop having contaminations and the CEO-toppling domino-effect will never happen.

Easier said than done, right? Well, it starts with hiring the industry veterans that have been there and seen it. (That's what Genzyme did to remedy the situation; several former colleagues are there right now).

Not sure which veteran to hire? Read our FREE case study to see if Zymergi's contamination-reduction expert is the right guy for you.

Stop Sucking At Contaminations