How Netflix Built Their Company Culture

Have a look at Netflix's Culture of Freedom and Responsibility.  It isn't right for every organization, but it is what most companies bloviate about.

Flip through these slides and salivate.

Culture from Reed Hastings

Intuitive Design per Jared Spool: Day 2 of @OSIsoftvCampus VCL#13

When I learned that Intuitive Design was going to be the 8:30am session

at OSIsoft's 2013 vCampus Live, I made sure to be on time. I've done a bit of reading on usability and plowed through Steve Krug's book, "Don't Make Me Think." (Why is the obvious so out of reach?)

Any corporate citizen who's had to operate enterprise software has had to deal with non-intuitive interfaces. From manufacturing execution systems (MES) to enterprise resource planning (ERP) systems, there are armies of confused employees relying on power-users to execute critical business workflows.

Why is it easier to purchase something from Amazon than it is to buy stuff using SAP? Why can I find information on Google faster than I can find trends in my data historian?

As with Steve Krug's book, I was not disappointed by Jared Spool's talk. I was quite stunned by his definition of "Intuitive design."

Mr. Spool says that there is such a thing as an acquired knowledge continuum (that he represented as an escalator).


Click on image to go to Jared Spool's site.

At the bottom of this continuum are people with no knowledge of how to use your system; at the top are people who know everything there is to know about the system (developers).

Somewhere on this continuum is what he calls, "Target Knowledge." Target knowledge is the amount of knowledge that the user needs to perform their duties. Also on this continuum is "Current Knowledge." Current knowledge is what the user knows right now.

The knowledge gap is the distance between target knowledge and current knowledge.

A design is intuitive when knowledge gap = 0.

Closing this knowledge gap is done in many ways. You can raise current knowledge with training. You can lower target knowledge with simplification. But to ultimately build an intuitive system, you at least have to know these terms and figure out at which point your system features impede usability.

The (lack of) usability with Enterprise Software is that enterprise software is feature-driven. As customers demand features that are required by their business, vendors clamor to pack their systems with features, in so doing raising the target knowledge higher and higher on the escalator.

And as MES and ERP systems evolve, they become unusable because only the few power users can master the richness of the features. Jared Spool's proposal is to understand your "killer features" and drop everything else.

by Oliver Yu

Jared's response to people who cannot summon the courage to drop features? "Don't worry, your competitors will."

He may be right. Here is Mr. Spool's website: User Interface Engineering (uie.com).

Thanks for the talk, @jmspool. It was a pleasure listening to you.

Automation Is War! (Wait, what?!?)

Unless you've been living under a rock, you would know that STUXNET refers virus that impacts automation control systems. It was first discovered at the Iranian facility that was enriching uranium. And since then, we learned that this virus wasn't created by hackers; the sophistication of it points to a government with vast resources. Check out this fascinating article on STUXNET.

Here are some main points:

• There isn't one Stuxnet, there are a set of twins.
• The first Stuxnet attack happened in 2007 and was designed to stealthily disable the uranium enriching centrifuges by overpressurization. Its impact is uncertain.
• The second Stuxnet attack was less stealthy and was designed to disable the centrifuges by over-spinning the centrifuge rotors.
• The second Stuxnet virus spreads and infected the laptop of system integrators that connected to the SCADA system and the intent appears to be to disable the Iranian nuclear program over the long-term.
• The forensics on the Stuxnet virus points to government involvement as the Stuxnet attack uses "zero day" Windows weaknesses and stolen digital certificates that would cost hundreds of thousands of dollars to create.

Here's something amazing:

One of the first things this Stuxnet variant does is take steps to hide its tracks, using a trick straight out of Hollywood. Stuxnet records the cascade protection system's sensor values for a period of 21 seconds. Then it replays those 21 seconds in a constant loop during the execution of the attack. In the control room, all appears to be normal, both to human operators and any software-implemented alarm routines.

Stuxnet can intercept sensor measurement and send false values to operators at the HMI. Operators at the HMI are none-the-wiser. OPC Servers have no idea that these values are counterfeit. OPC Interfaces, repeating what they were sent by OPC Servers, would simply re-transmit these false values. Data Historians, archiving what they are told by OPC Interfaces, would insert these false values in the archive as told.

The long-term effects are truly pernicious because long-term process understanding is impacted.  The power of data historian software is the ability to let users look in the archives and understand real phenomenon from sensor readings.  And since Stuxnet decoupled sensor transmissions from reality, there will be armies of engineers scratching their heads.

I'm headed to OSIsoft's vCampus next week and going to take part in the security hack-a-thon as SCADA security comes to the fore.  Most data historian run as read-only systems: data flow unidirectionally from the control system to the data historian, so for most systems, security impact ought not be that large for process data historians.

In the meantime, if you're a system integrator, do not connect to customer SCADA.  And if you're a customer, don't be letting system integrators connect to your SCADA.

Also, questions for automation purchasers to think about:

Have you assessed the risk of a stealth virus/worm infection like Stuxnet?

What risk-mitigation tactics are available to your automation team?

Unknown/Foreign Substance in Biologics Manufacture

by Oliver Yu

At small-scale, you get to use glass; and because glass is transparent, you get to observe transparent or slightly turbid solutions in glass miniferms or spinners.

Excluding single-use technology, large-scale is still done using stainless steel, and the problem with stainless is that it's not see-through, which means you don't get to "see" the process.

Because of this, biotech manufacturers come across unknown or foreign substances in the process. Years ago, I was called in to look at "murky black" substance that was settled at the bottom of an alkali solution holding tank. The tank was stainless steel and the alkali solution should have been transparent but there it was, an unmistakably dark cloud of something at the bottom of the tank.

Looking through the sight-glass, I ask the operator:

"Is this the first time you've seen this?"

The old-timers? No, but [I]? Yes.

"Do you see this [murky solution] when you make up a fresh batch?"

We're not really looking, but I don't think so.

"If we take a non-routine sample, is the sample going to be dark?"

No. It just looks like base solution when you get a sample.

In the GMP world, it's better to be safe than sorry. Decisions impact lots of bulk product. The product gets injected into patients so if you see something, even if it turns out to be nothing, you ought to say something.

We ended up taking a non-routine sample, opening up a discrepancy in the QA system and let the Quality Management System do its job.

In the end, the sample came back testing for no foreign substance. The visibly black smoky fluid was, by all analytical means available to us, the alkali solution that was supposed to be in the stainless steel tank.

And therein lies the problem with unknown/foreign substances in biotech manufacturing. You're going to find unknown white substances floating on the surface of your culture like an amoeba, but it's probably going to be antifoam. You're going to find unknown brown substances and it's probably going to be dried media.

You're going to boroscope the line and find crusty white stuff. You're going to find black substance. You're going to find all this stuff and the only reason it's unknown is because you don't have the process history or experience to know that it's actually normal.

Whether you take an ocean-boiling approach to identifying all the foreign substances you find, or whether you test a few here and there depends on the regulatory "flavor of the month."

However it is that you respond, make sure you can pass the red-face test when queried about it by external auditors that have the ability to legally shut down your operations.

And whatever you do, pass that red-face test in a way that doesn't open up new cans of worms.

FDA Is Conducting Fewer Inspections (Yayyy!)

by Oliver Yu

Zymergi's customers include large-scale biologics manufacturers: biotech companies that use cell culture or fermentation to make medicines. In the United States, biologics manufacturers are subject to regulations under Title 21 of the code of federal regulations, which authorizes the U.S. Food and Drug Administration (FDA) to inspect facilities.

Few take FDA inspections more seriously that biotech/pharma companies. Inspections are the path to facility shutdowns. Inspections lead to 483s. 483s lead to warning letters. Warning letters lead to consent decrees.

See FDA Inspections Funnel.

There appears to be relief for those in the regulated markets.  Data collected by FDAzilla (a Zymergi software licensee) shows that the FDA is conducting fewer inspections.  The number of inspections started declining in Q3 2012 and bottomed out in Q2ish 2013.
In fact, the FDA has conducted fewer inspections in 2013 than in any period in the 4-years prior.

It's not clear if biologics manufacturers have benefitted.  Just because the FDA is conducting fewer inspections does not mean that this decrease is impacting all areas (e.g. Food, Cosmetics, Tobacco, Drugs, Biologics, Devices...etc.) evenly.

Also, this decline in inspections appears to have little to do with the recent federal government shutdown:


That last bar (circled in red) is the shutdown.

Contamination Control of Cell Culture Bioreactors

by Oliver Yu

"Contamination Control"

A misnomer. I can see how they got that name... from "Pest Control," but I still hate it.

"Bioreactor control" makes sense as cell culture manufacturers try to direct the behavior of pH, dissolved oxygen, temperature, agitation, pressure...

But "contamination control"? No one is trying to direct the behavior of bioreactor contamination: Everyone tries to abolish bioreactor contaminations.

The abolition of bioreactor contamination in a large-scale setting is generally a team sport. It can take just one person to solve the contamination. But usually, the person who figures out what went wrong (the brains) is unlikely the same as the person who implements the fix (the hands). And in a GMP environment, the change implementation is a coordinated process involving many minds, personalities, and motivations. With all those people come an inordinate amount of politics for a goal that everyone seems to want to reach: no contams.

Immediate-/Medium-term Fixes

The first thing to realize is that operations management is usually the customer when it comes to solving bioreactor contaminations. Everyone's butt is on the line, but no group burns more resources responding to bioreactor contaminations than them. And in my experience, there is no "short-term" vs. "long-term" solution.

To operations management, there's just immediate solution and medium-term solution.

• Immediate solution :: what are you going to do for me today?
• Medium-term solution :: what lasting solution are you going to implement after the immediate solution?

Science... if it fits

The second thing to realize is that there's no room for science. The prime objective is to stop the contaminations. The secondary objective is to find the root cause. If identifying the root cause helps stop the contamination, that's a bonus; but root cause or not, you still must stop the contaminations.

For example, if your contamination investigation team thinks that there are five contamination risks, the directive from management will be to implement all CAPAs necessary to address all five risks. If the fixes work, "Great! You met the objective." Do you know what the true root cause was? Not a clue (it was one of those five, but you'll never know which one).

Political

The third thing to realize is that contamination response is as much political as it is technical.

• You can have the right solution, but present it the wrong way - and it's the wrong solution.
• You can formulate the right solution that is not immediately actionable, no one wants to hear about it.
• You can irrefutably identify the true root cause (thereby shining light on GMP deficiencies), and run against resistance.

Being right is different than being effective. And "bioreactor contamination control" at large-scale requires effectiveness. For in-house resources, it requires a keen understanding of interpersonal dynamics. For organizations that are at either a technical or political impasse, there are external bioreactor consultants who understand how to effectively troubleshoot and abolish bioreactor contaminations.

Abolish Bioreactor Contaminations

Post-Licensure Cell Culture Process Improvements

There's a great article out in GEN on cell culture process improvement, in particular, the Dr. Yuval Shimoni segment on the "low hanging fruits" of post-licensure improvements.

From the article:

At the CHI conference, Dr. Shimoni demonstrated how changes to cell culture media can make a difference by increasing production capacity through greater cellular productivity.

As I didn't go to the conference, I am left thinking that his feat was pretty impressive.  Changing media components post-licensure is quite daring.

The biologics license agreement (BLA) will call out the exact ingredients +/- percentages on each media component.  And changing a single component can (and has shown to) alter product quality.

Changing several media components, if in fact, that's what he did, is quite the feat and would take testicular fortitude of magnitude 10 on the Moh's scale: Any adverse impact on product quality - no matter the cell productivity improvements - is unwelcome.

Pulling off a media-change post-licensure is not only a technical accomplishment, but a political one as well.  

Data Visualization - Learn By Looking

by Oliver Yu

So the least rigorous form of statistical analysis is simply looking at the data. I've written about this before where you can tell quite a bit about a phenomenon just by looking at the data (no p-values, no alphas... just looking at the data)

Here was that example distribution of test scores:

When you look at the data and there are irregularities or non-smoothness, you're looking at some human intervention... some manual action that does not comport with the natural order.

Have a look at this visualization. It's apparently the average monthly premiums for insurance plans under the Affordable Care Act for each county. Dark blue are plans that cost $250/mo. Dark red are plans that cost $1,250/mo... so the more red, the more costly.


What's most interesting to me is that you can see the shapes of Virginia, Wyoming, South Dakota and New Jersey pretty well on this map according to the price of ACA insurance premiums. Some guy living in Montana is paying $500/month; cross an imaginary line into Wyoming and now it's $1,000.

When you see something like this, you can infer that a non-natural phenomenon holds the true explanation (e.g. state law). There's a step-function here and step-functions aren't found that often in nature.

Now have a look at New England: here, there's a gradient... the farther northeast you go, the more costly the insurance. Likewise in Wisconsin... the closer you get to Minnesota, the more expensive the premium. Gradual changes or smoothness is what we can expect for nature. And a lot of information can be inferred by just looking.

Unnecessary Testing in cGMP World

by Oliver Yu

In the world of soaring medical costs, we have unnecessary medical tests and procedures sapping precious healthcare dollars.

The thing about these medical tests is that they are necessary for someone under some circumstances, just not for most people under most circumstances.

So is the case in the world of GMP biologics manufacturing. There are

Image courtesy of the Chemical Heritage Foundation Collections.

plenty of tests that need to happen to produce a releasable lot. There are in-process tests; there of Certificate of Analysis (CofA) tests.  There are analytical tests you for engineering runs; there are tests you perform on contaminated lots, but not others.

But regardless of what test you are performing, the litmus test for performing the test or analysis is:

Does the result of the test help me make a decision?

Consider the following snippet from a computer program:

   if ( testResult == PASS )
   {
      forwardProcessBatch();
   }
   else
   {
      forwardProcessBatch();
   }

In this case, if I run a test and pass, I get to forward process the batch. If the test fails, I still get to forward process the batch. So if in either case, I get to forward process the batch, why should I bother doing the test? The result of the test does not do anything to serve the outcome!

Another good way to approach the question of whether or not to perform a test is to see if you can write down a plan for what to do with the test result. If you can write down a reasonable plan and stick with the plan prior to getting the test results, then there's a good reason to perform the test; otherwise, you're simply on a fishing expedition and making it up as you go along.

FIO Samples

There exists "For Information Only" samples that are specified into the process.  For example, concentrations of ammonium (NH₄⁺), sodium (Na⁺), pO₂ and pCO₂ are measures of cell culture metabolism that are useful for long-term process understanding.  They likely never be used to make a forward-process decision, though they can be used for retrospective justification of discrepancies or as variables during multivariate data analysis.

In my experience, these routine FIO samples are contentious.  On one hand, they serve the purpose of long-term, large-scale process understanding as well as sporadic justification for discrepancies.  On the other hand, if FIO samples get used enough to close discrepancies and release lots, over time the FDA and other agencies will pressure you into making these FIO tests into in-process or lot-release tests.

Defensibility

In the end, your actions in deciding to perform a test need to be defensible.  You need to defend the costs to do the test to management.  You need to defend not doing the test to the FDA.  And your situation may be different than the biologics manufacturer down the street.

That defense ought to rest on whether or not you can do something with the result of the test.

Oldie, but Doozy - Scientists are not Statistician

Here's something that just came to my attention: last year (2012), Amgen reported that they attempted to confirm the published findings of 53 "landmark" studies involving new approaches to targeting cancer or alternative clinical uses. Only six (11%) of these studies could be reproduced.

Adding a second clue to this puzzle, Germany's Bayer HealthCare tried to validate published pre-clinical findings and found that only 25% were reproducible... an abysmal rate similar to the Amgen finding.

This past week, UC Davis professor of plant pathology - Pamela Ronald - issued a retraction of foundation work her lab did in 1995 and wrote of the mistakes in Scientific American. In her case, there were two errors, one of which was simply labeling:

In this way, new members of my laboratory uncovered two major errors in our previous research. First, we found that one of the bacterial strains we had relied on for key experiments was mislabeled.

Incidentally, Dr. Ronald cites the lack of reproducibility that Amgen found.

It's gotten to the point where the Economist has two articles out on it:

1. How Science Goes Wrong
2. Unreliable Research: Trouble at the Lab

Statistical Dunderheads

In the second article on unreliable research there's a segment on researchers who lack statistical knowledge, designing experiments whose results do not pass statistical muster because "scientists are not statisticians." Their conclusion comports with my experience: an epidemic of statistical dunderheads in science. Researchers are choosing N based on the number of slots in the pilot plant or based on the capacity of the lab. Scientists not understanding the risks of Type 1/Type 2 error built into their design.

The first step is to get some statistical training. After that, it's on-the-job training and getting on the phone with someone who is qualified. But the way it is and the where we are headed is simply not acceptable. Data-based decision-making is at the core of science (and for that matter, biologics manufacturing).  Scientists may not be statisticians, but perhaps they ought to be.

p.s. - It's interesting to note that it is publicly-funded academic research that cannot be confirmed by private-sector firms and not vice versa.

Non-Essential FDA Inspections to Resume!

News reports indicate that the government shutdown is over.  "Non-essential" federal employees that were furloughed included FDA inspectors (euphemistically called, "Consumer Safety Officers").

Apparently, even the government admits that FDA inspections are... non-essential.

Regardless, I'm quite certain that the effect on the GMP operations around the world was that of pure joy. Inspection Readiness Teams could relax a bit.  Regulatory Affairs and QA managers didn't have to be on high alert (especially ones that hadn't been inspected in a while).

FDAzilla reports a significant drop in sales of 483 documents as RegA and Quality managers had no reason for last-minute research on the inspector that didn't show up (but will still collect backpay).

But now that paid vacation the furlough for FDA inspectors is over, it's time for the GMP community to get back on their toes and maintain inspection readiness.

Good luck.

Who Are You Guys, Anyway?

So, I asked for a report to study Zymergi blog readers, and here's where the biotech/pharma readers are coming from:

This is a veritable who's who of the biotech world.  Obviously, you aren't all customers, but when it comes to large-scale biologics support, cell culture and bioreactor contaminations, readers and customers find themselves in good company.

Thanks for reading.

Note: All logos/trademarks belong to the trademark holder and inclusion on this list is not an endorsement of Zymergi or vice versa.

MSAT to Automation, MSAT to Automation. Come in, Automation

by Oliver Yu

When I was running cell culture campaign monitoring and we were using PI to review trends to understand physical phenomenon, there were times the trends didn't make any sense.

After digging a little, we found out that the data was simply recording with too little resolution either in the Y-direction or the X-direction.

Here's a short blog post describing the words to say to Automation (as well as some perspective) to get some more resolution in your data.

Compression Settings

If the data seems sparse in the Y-direction (e.g., you expect to see oscillation but only see a straight line), it could because the compression settings are such that too much data gets filtered out. For OSI PI, there are two types of filter settings: (1) exception and (2) compression.

Exception is responsible for filtering out repeat data between the data interface and PI.

Compression is responsible for filtering out repeat linear data within PI (between the snapshot and the archive).

Every point attribute can be viewed from within PI ProcessBook. And if you find that your exception or compression settings are too wide, view them within PI and make a note of what they ought to be, then go on and tell your Automation team.

In my experience, you'll find a reluctance within Automation for changing the individual settings on points. Generally, there is a standard or a rule that is applied uniformly to the set of points. For example, you're using Broadley-James pH probes in both cell culture and purification and we (cell culture) ask for a 0.005 compdev on bioreactor pH probes, shouldn't the buffer prep pH probes also be set to 0.005 compdev?

Automation has to balance the tension between customer (your) needs as well as defensible system configuration.

Generally speaking, you're going to be asking for changes to compdev of excdev point attributes, and if you're asking for more data to be collected, you want these numbers to be smaller.

Scan Rate Settings

What if after improving compression to filter out less data you still find that there is not sufficient resolution in the data to observe the physical phenomena that you know is happening? Well, the only place left to check is in the scan rate of the data... sparseness of data along the X-axis.

A point's scan rate is set based on a list of pre-defined intervals in the data interface. The data interface is a piece of software that transfers data from the source (control system) to the destination (data historian). If the interface is configured well, it will have sensible scan rates:

1. Every second
2. Every 2 seconds
3. Every 5 seconds
4. Every 10 seconds
5. Every 20 seconds
6. Every 30 seconds
7. Every minute
8. Every 5 minutes

It isn't always like this, but very often you'll see these intervals. The scan rates are defined in the interface configuration file and once set, they rarely change. The way it works is this, the first entry in the interval configuration is gets assigned: 1... the second entry: 2... the third entry: 3.

And whatever you set the point's location4 attribute is what it's scan rate is.

So suppose 00:00:05 is the third entry. Then a point whose location4=3 has a scan rate of every-5-seconds.

In a lot of cases, you simply tell your PI administrator you want the scan rate to be "every second," after which he's on the hook for looking up that scan rate in the interface. But FYI, if they said they made the change to the point but the location4 attribute is the same before and after, they're just BSing you.

There are a lot of considerations that need to get balanced when figuring out this stuff.  What's working against you is the default settings that come out-of-the-box with PI... as well as a generation taking the path of least resistance.

Get FREE Paper on PI Compression

Pros and Cons for Outsourcing Process Development

by Oliver Yu

I visited a contract development and manufacturing organization (CDMO) a few months ago.

What CDMOs do is they develop the process for you. And if you choose, they'll also execute the manufacturing process for clinical or cGMP material.

In short, if you're outsourcing either process development or manufacturing, CDMOs are the people you outsource to (pardon the preposition-ending sentence).

Dan Stark, currently senior director in the Global MSAT group at Genentech, once said (paraphrased):

The winners of the age of biotechnology will be the people who can translate research discoveries into commercial product and those who already have the infrastructure will have a head start.

What he's saying is if you're just a research outfit and you come up with new molecular entities, you're not in the position to extract the full value of that NME. Say you're IDEC pharmaceuticals and you come up with Rituximab... unless you can get the product into the hands of consumers, you're just a research outfit.

Likewise, if you are a contract manufacturer like Lonza where you have the manufacturing know-how and can produce clinical or commercial material, you also are not in the position to extract the full value of your know-how simply because you rely on someone else for your pipeline. In a lot of ways, you're a commodity (see Samsung getting into the biologics manufacturing space with Samsung Biologics).

Dan is saying that the people... the companies who can take a NME through the clinic... through the FDA approval process and be able to make the drug product are the ones who can extract the full value.

So why would anyone with a NME outsource the crown jewels of their business and go with external process development?

It turns out there are a lot of reasons.  To be sure that we got it right, Zymergi has collaborated with Dr. KC Carswell Ph.D of Carswell Bioprocess Consulting to produce a whitepaper on outsourcing process development.

If you're thinking about sending your process development into the hands of a 3rd-party, you definitely need to read the pros and cons of outsourcing.

Get FREE Whitepaper on Outsourcing PD