Public Service Announcement: FDAzilla rolls out InspectorRank

What's the first thing that happens when an FDA inspector shows up in your lobby?

by Oliver Yu

Personally, I don't know since I'm not usually hanging out in the lobby...

But when I was a cGMP citizen and I had to carry around a pager (smart phone these days)... everyone's pager goes off saying that "the FDA has arrived."

The element of surprise evaporates within 5-minutes of an FDA inspector showing up at your doorstep.

A conference room gets prepped to host the inspectors... A "war room" gets prepped as a proving ground for QA to review presenter talking points.

Good RegA departments will have intelligence on the inspectors already. Smaller outfits will do their research right then and there.

And a good website to go to is FDAzilla's InspectorRank pages. For example, when researching the FDA investigator for the Rhode Island bioreactor contamination, Megan Haggerty came up in the 483.

The InspectorRank page has her email, phone number and tells me right off the bat that she's based in Massachusetts, has audited 105 facilities since 2008 and inspects with a diverse group of peers.



Her historical 483 issue rate is 77%, so if Megan shows up at your door... you're probably going to get a 483 (yes, I know, this isn't a univariate problem).

Farther down, you'll see firms inspected in the past year. So for Megan... that's Amgen, AstraZeneca, Biogen, Genzyme, Alexion, Boehringer, Abbot... etc.


These are all the major biologics manufacturers, so if you're a biotech company, chances are... you're not going to be able to snow Megan.

One beef is that this data seems to be refreshed monthly, so it can be a little stale.

And as previously mentioned, you can even go and purchase select 483 documents that the inspector has issued.

If you have armies of Regulatory Affairs and Quality-folk on it, you're probably fine.  If you're a small- to medium- sized outfit, this may be a good resource for you.

Disclosure: FDAzilla licenses Zymergi software

Best Floor Drain Design for Bioreactor Sterility

So back to our question of which floor drain design is better for bioreactor sterility...



On the left, we have piping that is contained.  When fluids from the process piping go to drain, there is 0% chance that this fluid (e.g SIP condensate, or CIP rinse, or unspent media, or whatever) ending up on the plant floor.

On the right, we have piping that stops above the drain and there's a possibility that process fluids end up on the plant floor.

Well, the answer to the question, which configuration (left vs. right) is better for mitigating bioreactor contamination risk is..... right.  And here's why:


Should the process piping draw a vacuum--say via the Venturi Effect or with collapsing steam, the sealed piping allows for the contents of the drain to return to the process (as depicted by the brown arrows pointing up).

On the right, should the process piping draw a vacuum, only air from the plant floor (as depicted by the light blue arrows) is drawn into the process.  While still not ideal, air on the plant floor of a biologics facility (that is changed at some specified interval) is much cleaner than whatever you've been dumping down your drain.

You drastically reduce your chances of bioreactor contamination abiding by the uni-directional-flow-of-materials-through-the-plant principle.

by Oliver Yu

Not every manufacturing operations group has the luxury of good plant design.  Not many organizations have the resources to fund a greenfield plant starting with scratch paper.

As the executive vice president of that biologics manufacturer who was dinged with the 483 and a warning letter found out, these contamination and bioburden issues have a tendency to escalate.

The only way to get ahead of this escalation is to be proactive and apply prophylaxis.

With respect to microbial contamination of bioreactors, an ounce of sterility consulting is worth a pound of finished product.

Read how our consultant solved this string of 5 contaminations

Floor Drains Causing Bioreactor Contaminations?

On August 6th, FDA investigator - Megan Haggerty - issued this 483 to a Rhode Island biologics manufacturer. The first observation was about bioreactor contamination:

1. The firm's investigations into Soliris █████ microbiological contamination events are inadequate.

On August 22nd, the biologics innovator responded. It's strange because the response is marked, "Confidential" yet the FDA released this to the public via their website.

Dissatisfied with the response, the FDA issued a warning letter on March 22, 2013.

And this was such a big deal that management had to address contamination issues at the Q1 2013 conference call. When queried by an analyst, management said:

Let me start by saying, yes, we have identified the microbial agents that were the contaminating factors. They're soil-based organisms. We believe that they were probably introduced from someone's shoes, clothing, hair and probably related to an improperly functioning floor drain that has now been reengineered. And as I mentioned also on the call, we've also changed the way we clean and store our bioreactor equipment. And I think we have now -- we believe we've adequately addressed both of those issues.

by Oliver Yu

I can't say for certain that this was their problem, but we're seeing a lot of this process piping going directly to drain.  See the (badly hand-drawn) illustration below.  The left drain pipes process fluids directly to drain.  The right drain has the process piping that stops just above the drain.


Question: Which is floor drain design is better with respect to mitigating bioreactor sterility?

James T. Kirk, Plant Manager

If you think about it, the starship Enterprise is a plant (i.e. factory).

It's a plant that manufactures light-years.


Construction of the USS Enterprise NCC-1701 as depicted in Star Trek 2009

Kirk is the Plant Manager.

Spock is the Director of Technology.

McCoy is charge of EH&S.

Scotty is Director of Production (i.e. running the warp drive that produces all those light years).

And the SCADA (supervisory control and data acquisition) system is what they call the Enterprise's "Computer."

Nothing illustrates this better than this one scene from J.J. Abrams' 2009 reboot of Star Trek.

SPOILER ahead... If you haven't seen it, you should stop reading this post and go rent it on Amazon.

Then, you can go look up movie times and get tickets to the sequel(out this week).






Anyway, at some point in the movie, Kirk and Scotty get beamed aboard the Enterprise, but end up in utilities. Scotty gets beamed into the piping so Kirk has to go free him.


Where does he go?



Looks like an HMI (human machine interface) to me....



What's he doing? Oh, manually overriding a valve.



It's hardly recognizable as an HMI with those sexy lights across the top and snazzy faceplate graphics.



I guess they covered basic SCADA training in Starfleet ensign training.

But make no mistake. That looks like either a PLC (programmable logic controller) or a DCS (distributed control system).

And this is just for the utilities. The control system for the entirety of the Enterprise would be far more sophisticated.

I keep reading about how long we have to wait before we get Star Trek technologies... or how long before we have hoverboards...

But the fact of the matter is this. So long as we are minting CS and ChemE grads whose purpose in life is to get internet users to click on ads (as opposed to them creating and deploying SCADA software), it's going to be a long, long time.

At Zymergi, we're doing our part, furthering the deployment of these technologies by helping install, validate, and use these SCADA systems to manufacture biologics.

How about you?

Other reading:

• Why Isn't Automation Sexy?

All screenshots are from the Star Trek 2009 movie from Paramount Pictures, Spyglass Entertainment and Bad Robot Productions.

Industrial Centrifugation demonstrated on YouTube

For large-scale cell culture, centrifugation is one method used to harvest the production cell culture.

Before you can purify your active pharmaceutical ingredient, you need to get rid of the cells (yes, after all that work secreting the API, we get rid of the cells).

Note: some APIs are not secreted, in which case you want to keep the cells, so harvest means different things in different processes.

To separate the cells from the cell culture fluid (CCF), you can filter it or centrifugate it. If you need to process 12,000L or 15,000L in hours, you're doing tangential flow filtration (TFF) or you're using a centrifuge.

If you're using an industrial centrifuge to clarify cell culture, chances are, you're using Alfa Laval.

Three minutes of your time is all it takes to see how large-scale centrifugation works.



Nice work, Alfa Laval, Westfalia

Automation Engineer's Take on Wall-E

by Oliver Yu

I was talking with a buddy from my Cornell ChemE days (who now works in social media) about the odd trajectory of his career. Having had a successful career in biopharma and hospital administration, he's now a social entrepreneur. And it puzzled me that he is fulfilled "not using his degree" in social media.

From his side, he was puzzled that I liked running an automation business helping people get and interpret machine data so their factories operate more efficiently.

As an MBA, he explained, "Business is about people and relationships. I want operate in a world where people matter, and that's what 'social' is."

I have no disagreements with that statement. I did add:

Business is about making money...creating wealth. A world where everyone is wealthy is one where no one has to work; in that world, we have machines at our beck and call. Automation is the means to that world.

Screenshot from Disney Pixar's WALL-E where we find humans have fled Earth in a galactic cruise ship where no one has to work because their life is 100% automated.

Pixar's writers pose the question: What does the world look like when no one has to work?

Don't let Pixar's distinctly American interpretation (out-of-shape, chair-loungers watching TV while robots get us our beverage) distract from the world where everyone gets to enjoy leisure and no one has to work.

Some will jump in and say, "See, employment and working is good for man, else we'll end up all fat and lazy." It's true that some will choose this path, but the vast majority of others would do something else with all that time.

No truer words were spoken when man first uttered the phrase, "Time is Money."

Having vast wealth is synonymous with having vast amounts of time to do what you want; this time to do whatever we choose is called, "leisure." And the purpose of an economy is to lift as many of us from the bonds of employment as efficiently as possible.

As an aside, it's rather hilarious that our politicians run around trying to decrease unemployment. The world where everyone has the luxury of 100% leisure is a world where unemployment is 100%.

And all this leisure can only be possible because we created the machines to automate the tasks that would otherwise be manual.

But back to my buddy: he's also right. Ultimately, business is handled with strong personal relationships. And even after we've automated ourselves into a world where no one has to work, we'd probably spend all that leisure time socializing anyway.

More general commentary:

• Cell culture engineers are Keynesian

Bringing In The Big Guns

In the biotech world, perception can be (and often is) reality. The perception that you brought in the big guns matters as much as the reality that you brought bullets.

It actually reminds me of that scene from the movie, Men In Black where Tommy Lee Jones' character is arming Will Smith's character with this puny gun:

In the movie, this gun is called the "Noisy Cricket" and turns out to pack an enormous punch. However, the gun's small sizes makes it look like, at best,  a weak weapon.  Look at Will Smith's face.

What does this have to do with bioreactor sterility or cell culture consulting?

Well when you have a rash of contaminations or when your production campaign is on track for failing to meet ATP, you have actually TWO problems:

1. Solving the actual problem
2. Looking like you're solving the problem

A customer once told me, "Oliver, I need both action and PERCEIVED action."

He's right, and here's why:

by Oliver Yu

In claiming to have solved a problem, the first thing people will usually ask is how you solved the problem. If you tell them the solution and the solution is credible (perceived to be viable), then great, problem solved and let's move on.

If you tell them the solution and the solution seems dubious even though it worked, you will get lingering questions and second-guesses.

Managing the perception is as important as managing the reality.

Play-by-play of Microbial Contamination Response

As indicated in a previous post, the classic sign of contamination observed in your dissolved oxygen (dO2) trends.

But how does this play out in real-life?

In my experience, the front-lines are the ones detecting contamination.  Your operator is typically the first person to see alarms on the HMI.

For cell cultures, they're going to see that the dissolved oxygen (dO2) is at 0%.  That pH has also "crashed" to a low number.

In response to these signals, the dO2 controller is max'ed out at 100%, and the pH controller has the peristaltic pump for sodium carbonate (alkali solution) spinning.

The operator (probably on a walkie-talkie) calls out to the front-line technical support group, such as manufacturing sciences campaign support group, to verify; at which point, they'll pull up a trend on PI and see something like this on the dO2 trend:


If the trends look like a contamination, he can recommend that the bioreactor contamination response SOP be executed.

Hemocytometer
courtesy of Jacopo Werther

The bioreactor contamination response starts with pulling a sample and verifying under a hemocytometer as well as sending the sample to QC.

Most quality control have a microbiology department with guys who run the validated tests for detecting contamination or bioburden.

If foreign microbes are confirmed in the samples, the bioreactor decontamination response is executed to inactivate the microbes and send them to the waste-kill pit.

The contamination response SOP also mobilizes all the relevant departments: Production, QA, Manufacturing Sciences, QC, Facilities.  All for one of those politically-savvy contamination meetings.

No one likes having to go to one because these meetings are a huge bummer.  The company just produced scrap.  A bioreactor is out of commission.  And your workload just increased for this "non-routine" item.

From here, you pick your favorite root-cause analysis tool and run through the system.

Our experience is that this tends to work if your contaminations aren't back-to-back.   Biotech firms can endure the sporadic contaminations, but if you start getting back-to-back contaminations, faith in the existing contamination response goes out the window and people start looking for 3rd-party consultants.

You call your network,  you search the internet,  and you end up reading blog posts by some company that's trying to sell their contamination consulting services.  You might check LinkedIn to see if you have any connections in common.

You might download a microbial contamination case study and see if they've solved some tough problems.

Then, you might want to call them, but when you do, you can't say too much until you get a non-disclosure agreement going.

After that, you're free to send them information while they prepare a quote for services.

They come in, see things you haven't seen, write up a report, and boo-yah, life is good again.  At least that's how it goes if the third-party consultant is Zymergi.

Basic Microbial Bioreactor Contamination Probabilities

by Oliver Yu

Before my time at Cornell, I heard there was a professor who gave a grade based on the product of a test score and a lab grade.  If you got a:

• 10 on the test and a 9 in the lab, your score: 90.
•   9 on the test and a 9 in the lab, your score: 81.
•   8 on the test and a 7 in the lab, your score: 56 (ouch!)

When you get into mathematical situations where the final outcome depends on the product of two numbers being high, you're in a tough situation.

One time, I got tasked with figuring out what the probability of any natural disaster striking a biologics manufacturing plant given an estimate of the individual probabilities of said natural disaster.

So suppose:

5-year probability of disaster happening

Disaster	Probability
Earthquake	5%
Grass fire	10%
Flood	2%

The key to answer this question is thinking in terms of "not."

5-year probability of disaster NOT happening

Disaster	Probability
Earthquake	95%
Grass fire	90%
Flood	98%

So what's the probability that nothing happens?

0.95 x 0.90 x 0.98 = 0.838

Even though your probabilities of the individuals are in the 90% range, the probability that not any of them happen is in the low 80's. You have an 84% chance of nothing happening, which means the probability of something happening is 16%.

So the equation is thus:

1 - ( 1-p₁ ) x (1-p₂) x ... x (1-p_N)

This is one of those huge mathematical bummers... the more things that can go wrong with your process, the success rate odds are stacked against you.

The same is exactly true with contaminations.

If a successful run depends on five separate operations. And the failure rates for those five operations are:

• p1 = 1%
• p2 = 1.5%
• p3 = 2%
• p4 = 2.5%
• p5 = 3%

Then success rate of the system is:

99% x 98.5% x 98% x 97.5% x 97% = 0.90%

Five measly steps each with failure rates less than 3% and your overall failure rate is 10%.

Next time you're troubleshooting your microbial bioreactor contaminations, think about this math. If your culture success rate is 10%, your status-quo aseptic practices can be executed >97% of the time and you can still contaminate 1 in 10 runs.

Don't wait until crisis mode.

Further reading:

• How to detect bioreactor contamination
• FREE Case Study on solving bioreactor contamination
• You Suck At Bioreactor Sterility

Amgen's Biosimilars Offensive

Here's an article at Genetic Engineering & Biotechnology News on the top 10 innovator drugs competitors are trying to copy.

1. Amgen's Aranesp
2. Amgen's Enbrel
3. Amgen's Epogen/Procrit
4. Pfizer's Genotropin
5. Roche's Herceptin
6. AbbVie's Humira
7. Amgen's Neulasta
8. Amgen's Neupogen
9. Janssen's Remicade
10. Roche's Rituxan

This list appears to be alphabetical (by the drug's trade name).  As you can see, 5 of the top 10 biologics that are marching inexorably towards patent expiration belong to Amgen.

It's no wonder why Amgen needs to start playing offense.  And it's no wonder why they need to start now when the regulatory pathway for biosimilars is still a work-in-progress (WIP).

I'd like to see these companies get more lean and compete on manufacturing agility as well as efficiency.

Processes need to be better specified... the operating space ought to be better characterized (QbD).

Once the technology is transferred to large-scale, there ought to be a program to continuously improve these processes to eliminate variability and increase process understanding (PAT).

All of this manufacturing sciences and technology is already happening, but I have yet to see companies trot it out as a core competency.

It's not sexy, but in a world where your top-line revenues are in decline, focusing on bottom-line numbers is one of many viable paths.