New Features in ZOOMS 2.2.3 (which has shipped, btw)

Yep, ZOOMS 2.2.3 is shipped. There are multiple upgrades, including several under the hood, but the user-features are as follows:

Autocomplete Detects Text Within Words

We are fast learning from customers that despite S88, people naturally like having the equipment hierarchy in the unit name. It's natural and logical. As such, we've modified our autocomplete to search after delimiters such as the period (.) and the underscore (_). You can see below that the user typed "10" and yet items that have "10" after a legal delimiter show up.

TimeExpressions

Inputting time is a pain in the buttocks. ZOOMS already recognizes the PI time expressions (*, t, y) as well as (60s, 1m, 1h, 2d, 3w, 4y...) time intervals. But our users like not having to type the forward slashes (/) and colons (:) that come with inputting time like "11/23 23:00."

ZOOMS now recognizes several more oddball time formats that don't come standard. You can see below that "1123 2300 to 1130" resolved to: 11/23/2014 23:00:00 to 11/30/2014 0:00:00.

Downloading Raw and Evenly Spaced Data

The ability to download raw data has existed since 2.2.2. Raw data is the actual data captured in the data archive. Unfortunately, the captured data may not happen at the same timestamp and if you need to correlate values with other parameters. So for version 2.2.3, we have the ability to download evenly-spaced data in user-specified increments. Click that "timed CSV" button and you're going to get interpolated data values every 15 minutes:

Trend data visualization and data downloads remain a core offering.  Stay tuned for more release announcements.


Request a ZOOMS demo

Could not load file or assembly 'System.Data.SQLite' or one of its dependencies.

This is why you're here:

1. You're getting this error.

   <add assembly="System.EnterpriseServices, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"/> <add assembly="System.Web.Mobile, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"/> <add assembly="*"/> <add assembly="System.Runtime.Serialization, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089, processorArchitecture=MSIL"/> <add assembly="System.IdentityModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089, processorArchitecture=MSIL"/>

2. You're on a Windows OS
3. You're trying to connect to SQLite (System.Data.SQLite.dll)
4. You're trying to get an IIS/ASP.NET web application to connect to SQLite
5. You're in luck.

This a pretty simple fix that we talked about before.

What you need to do is Enable 32-Bit Applications. 



When you Enable 32-Bit Applications (by setting it to true), the worker process(es) serving the application pool will be in WOW64 (Windows on Windows64) mode... which means that your application pool is now a 32-bit process that loads only 32-bit applications.

Metadata Nomenclature for ZOOMS units and parameters

So in the world of industrial automation, there's this S88 standard for batch.  And while it doesn't address everything, it is an oldie, but goodie.

S88 talks about an equipment hierarchy, which is a way of saying:

Your equipment can be named like so that each unit is unique.

Here's an example:

Suppose you're standing in your plant looking at a your three production fermentors: F101, F102, and F103

And let's suppose that they are located in the Fermentation suite of your brewery.

Let's say your brewery is in Davis, California and the name of your company is Zymurgy Brewing.

Well then one way to name that production fermentor: F101 is as follows:

/Zymurgy/Davis/Fermentation/F101

and for the rest of your units:

/Zymurgy/Davis/Fermentation/ProductionFermentor/F101

/Zymurgy/Davis/Fermentation/ProductionFermentor/F102

/Zymurgy/Davis/Fermentation/ProductionFermentor/F103

Now let's suppose Zymurgy acquires a brewery in Dixon, CA with the same unit names: F101, F102, F103, but at that brewery, those units are portable tanks:

/Zymurgy/Dixon/Fermentation/PortableTanks/F101
/Zymurgy/Dixon/Fermentation/PortableTanks/F102

/Zymurgy/Dixon/Fermentation/PortableTanks/F103

By identifying units with their hierarchy, it's very difficult to get "collisions" where two units with the same name, have the same exact hierarchy.

Now it's easy to declare "F101, Agitation" or "F103, Air Flow"

Easy right?

Well suppose you didn't want to name your units that way.  Suppose you want the unit to be:

DIX.F101

DIX.F102

DIX.F103

That's fine, too... at least within ZOOMS.

The next hiccup with metadata nomenclature is trying to figure out which part of the descriptive goes in the unit vs which part goes in the parameter.

For example, suppose you had a water chiller:

VV.CHILLEDWATER4000 could be the unit and Temperature could be the parameter.

Alternatively, you may go with:

VV.CHILLER4000 as the unit Water Temperature as the parameter.

Should "Water" be in the unit name or should it be in the parameter?  

There's no hard rule... only that the unit and the parameter should have high reuse.   This is to say, if you have multiple homologous units, yet one is "Temperature" while the other is "Water Temperature," there's probably a better way to handle your naming.

Could not load file or assembly 'X' or one of its dependencies.

You're seeing this Configuration Eror:



And you're using IIS7 with ASP.NET and using 32-bit applications.

Chances are, you just installed a fresh copy of IIS and you haven't changed around any defaults.

You're in luck because the fix is easy.  Here's what's going on.

The default installation of IIS 7 is to disable 32-bit Applications.  Afterall, IIS 7 comes with Server 2008 and Server 2008 is an x64 system.

So all you have to do to get rid of this error is to Enable 32-bit Applications in IIS.  Here are the steps:

1. Go to Start and type in inetmgr to launch IIS
2. In the left page, select Application Pools
3. In the middle pane, select the application pool corresponding to the web-application that is giving you this error.
4. In the right pane, select Advanced Settings:
    
5. This is what you'll see: Enable 32-bit Applications is set to False.
   
6. Go ahead and set that to True:
   
7.  Click OK

And you're super awesome application that depends on 32-bit DLLs is back in business:

iOS 8 Safari Cannot Access Windows Authentication IIS websites

So our flagship product, ZOOMS - search engine for OSIsoft PI - is a web-application... meaning that like Google, you get to the app by using a web-browser.

ZOOMS itself is browser-agnostic - we don't care what browser you use, just that you use a modern browser.  It's HTML5 meaning that all modern browsers ought to be able to view it.  And all modern browsers include Safari on your mobility devices.

ZOOMS is also built on the Microsoft stack, meaning that the operating system is Windows Server.  The web-server is Internet Information Services (IIS).  The database is OSIsoft PI.  And the application programming language is ASP.NET.

For customers using Windows Active Directory, being able to authenticate users against the AD is useful and so that just means that before you get to use the website, you have to type in your active directory username and password.



Imagine my surprise when I heard from a customer that "ZOOMS isn't working."

After a lot of troubleshooting, it turns out that iOS Safari doesn't play well with Windows Authentication.

When trying to access intranet sites in our company that use Windows (IIS) Authentication, it challenges for the login and password 2 or 3 times, then nothing - just hangs. These sites work fine on iOS 7 devices - and worked on the same devices that were upgraded. Windows Authentication was broken in iOS 7.0 also and not fixed until a later update. Why does this keep getting broken?

This was exactly what happened to my customer.  Gets challenged for their password twice and then the page hangs at ~20%.

For now, ZOOMS users should understand that iOS 8 is unsupported - not because of ZOOMS, but because of iOS 8 - and that if your situation is feasible: disable Windows Authentication until Apple comes up with a fix.

Mathematical Bummers and the American Canyon/Napa Earthquake

As Northern California cleans up from the American Canyon/Napa Earthquake of August 24th, 2014, I am reminded of a mathematical exercise I was asked to perform in my early years as an engineer with my former employer.





The question posed by upper management was this:

Given estimated probabilities of an earthquake, grass fire, flood, etc... what is the probability in any given year that any of the natural disasters hit the factory.  For example, suppose estimates from the U.S. Geological Survey for the probability of natural disasters in any given year were as follows:

p_earthquake = 5%
p_{grass fire} = 10%
p_flood = 2%

What is the probability that any of these hit?

Well, if you multiplied the probabilities, you'd simply get the chance that all 3 of these natural disasters happen in the same year... so 5% * 10% * 2% is 1 in 1,000.

But that was not, "What is the probability that all of these hit in the same year?"  The question is what is the probability that any of them hit in the same year?

To get to the bottom of this, you simply ask yourself, "What is the probability that none of these things happen in any given year?"

Well,
the probability that no earthquake hits is 95%.
the probability that no grass fire hits is 90%.
the probability that no flood happens is 98%.

So the probability that none of these things happen is 95% * 90% * 98% = 83.8%

What's the probability that any one of these things happen? 16.2% a.k.a 1 in 6.

So we handed this answer into upper management and I wasn't in the room, but I was told that my director got scoffed at for presenting this figure.

It's been 13 or 14-years since I handed in that calculation and they've been rolling that dice every year.  Now that it's happened (I slept through it, "shaken, not stirred"), I feel a bit vindicated.

Why does this matter to Zymergi or our customers?  Well, it's the basics of bioreactor contamination probabilities.

If you add up the probabilities of your sterile envelope failures as well as your sterile operations failures, you end up with an N-sided dice that you roll each time you put up a production culture.  And if you've done the math, you realize it doesn't take that many low probability vulnerabilities before they add up.

Process Data for the Rest of Us

I first started using OSIsoft's PI in 1999 when I got hired as a fermentation engineer at the world's largest cell culture plant (at the time, 144,000-liters).

I remember the first week vividly.  Me - not knowing anything about how read cell culture trends - watching my boss run his cursor over PI trends, nodding his head and then running off to meetings telling the operations staff what's happening in the bioreactors.

Over the years, I've put my eyeballs on thousands of cell culture runs and became an expert on the matter.  Yet, no matter how many trainings gave to tech(nician)s, supe(rvisor)s, and managers to increase the population of process experts at the company, I got the sense that ultimately it was still my team's (i.e. manufacturing sciences') job to know what's happening to the cell culture runs in-progress.

OR, maybe not...

The VP of the plant had the PI administrator write a script to open up PI ProcessBook and snapshot current trends as a images and put it on a website (remember, this was back in 1999).  Clearly management recognized the value of these trends, but was just too much activation energy to get PI data to "The People."

So when I left my awesome company (and awesome job), I set out to do one thing:



And by PI, I mean "process data."

Google had already pioneered the one text-box, one click format for accessing website data.   So why is it that cGMP citizens can find information on the internet faster than they can find critical process data from within the enterprise?

This is why I bothered creating ZOOMS and this is why I think that there's a place for ZOOMS in the competitive space of trend visualization on the web.

It actually wasn't until the Jared Spool lecture at this year's OSIsoft PI User's Conference that I learned how to better enunciate this creation.  


A quick recap:

The bottom of the escalator is where you are if you know nothing about an app (new users live here).  The top of the escalator is if you know everything about the app (developers live here).

Current knowledge is where the user is today; and target knowledge is where a user needs to be to know enough about the application to perform his duties.

Mr. Spool tells us that intuitive design happens when the knowledge gap (the difference between target knowledge and current knowledge) is zero.

A key observation is that the more powerful and feature-rich the tool, the higher up the target knowledge is...and the harder the knowledge gap is to close.

The success of Google (which is to be replicated with ZOOMS in the process trend visualization context) is a modest number of features in order to lower the target knowledge... and thus diminish this knowledge gap and achieve intuitive design.

There are plenty of feature rich PI trend visualization tools for process experts.  ZOOMS is process trends for the rest of us; in other words: PI for the "non-user."

At the end of the day, it's People, Process, and Technology... in that order.  You can buy awesome technology, but if only a small minority of your people use it, you're neither capitalizing on their potential, nor your process.

ZOOMS v2.2 shipped!

by Oliver Yu

Wow, what an exciting week.

ZOOMS version 2.2 is shipped! The Top 3 features in this release are:

• Wildcard tag search
• Digital States now rendered; Step=1 recognized
• Full-text search on units

Wildcard Tag Search

Wildcard tag search was a feature that was added back to ZOOMS. This feature originally left behind because the idea behind ZOOMS is that users want to search metadata, not tags. As it turns out, there's a delay between when the tags get added to the data archive and when the ZOOMS metadata gets updated, and in between that time, these tags aren't findable on ZOOMS. Well, we listened to our customers and it's back in there.


ZOOMS detects if there is a wildcard in the query; if there is, ZOOMS will do a tag search and show the results asynchronously.  This way, the slowness of the data archive does not impact the speed of the page load.

Digital States and Stepped Tags

Up until now, ZOOMS interpolated all values - which is great for a majority of numerical tags that come in at high frequency. But what about the tags that don't change in value... or are monotonic in nature (like setpoints or OFF/ON)? Well, the glitch in ZOOMS prior to this release is that they'd be plotted as interpolated and didn't make much sense.


 This glitch is fixed.  In the above image, the magenta tag is stepped.

Full-Text Search on Units

This was another great suggestion from one of our best customers. Prior versions of ZOOMS could identify the name of the unit; and therefore suggest it as a user typed it and combine it with parameters to produce a trend. The problem is users often know about the unit by its description or where it is in the equipment hierarchy... not necessarily by name. With this upgrade, any word that appears in the description or hierarchy of the unit gets recognized by ZOOMS and will show up in the SERP.

In the image below, you can see that the user is interested in the 12,000-liter bioreactors in the plant and has typed in his colloquial search term, "12kL."

ZOOMS will return the units T440 through T447 as that "12kL" string appears in the descriptor (but not the name).



Incidentally, in the above query, user could have searched on "ZYM", "CCP1", or "Fermentation" and gotten results.  Now, the user has to know even less about the equipment to get a search result.

Previously added features:

• Ability to magnify the x-axis
• Ability to scale the y-axis
• What is semantic search for process data?

How to Scale X-Axis for ZOOMS Trends

by Oliver Yu

In the last post, we talked about scaling the Y-Axis of ZOOMS trends.  What about the X-Axis?

Well, the X-axis (or as pedantic mathematicians would call it, "abscissa") for ZOOMS is always the time-axis.  As with any chart, there is an "x-min" for the minimum value of x as well as an "x-max."

In the world of time-series data, there are more colloquial names:

• x-min is starttime
• x-max is endtime

Unfortunately, there are also hundreds of ways of representing time:

• 04-Jul-1776
• 12/8/1941
• 

Type Time Range into Search Box

And typing in the starttime and endtime is one way to set the time range:

ZOOMS is able to interpret a lot of date inputs, but not all of them.

Use the Trend Buttons

What happens when you've got a trend and you want to go from there?
There are 3 buttons at the bottom of each trend.

1. Back Arrow is an arrow pointing to the left that will take the trend one time-range into the past.
2. Forward Arrow is an arrow pointing to the right that will take the trend one time-range into the future.
3. Refresh/Revert will restore the time-range to the original as-loaded trend.

Highlight Area On Trend

You can magnify a time range by highlighting it with the mouse: clicking and holding down the mouse button at one edge of the time range and dragging the mouse to the other edge of the time range:
When you release the mouse button, the trend will zoom to the selected time range.

Summary

In summary, there are at least 3 ways to scale the X-Axis of a ZOOMS trend:

1. Type in the time range in the search box.
2. Use the "back" and "forward" buttons
3. Highlight an area on a trend

See Also:

• How to Scale Y-Axis of a ZOOMS Trend
• Controlling Who Can View Process Data with ZOOMS
• OSIsoft PI Search Engine: ZOOMS

How to Scale Y-Axis for ZOOMS Trends

by Oliver Yu

There are several improvements with ZOOMS 2.1 and the first difference you'll notice is the trend controls... the buttons at the bottom of each trend that "control" what happens to the trend.

For this post, we'll be focusing on the Y-axis scaling buttons, which are 5 o'clock position when looking at the trend from straight on.  There are 3 buttons:

• multi
• channel
• single

Multi

By default, every trend is plotted as "multi," which is an abbreviation for multiple Y-axis.  Each trace on the trend gets its own Y-axis scale where the bottom is the minimum value and the top is the maximum value:

While you get a fuller sense of each trace - since each trace spans the entire range of the graph, there is no sense of what the traces are like in absolute value.

Single

The left-most button is "single," which refers to plotting all the traces on a single Y-axis. Using this method, you can get a strong sense for how the traces relate to each other.
Compared to the multiple Y-axis, we can now tell that the orange sinusoid is oscillates in a lower range than either of the red or magenta traces.

Channel

The channel Y-axis scaling divides up the trend in N horizontal slices and plots each trace in its own channel.  Using this method, you can hover your cursor over the entire trend and get a feel for what the values actually are.


There are plenty of awesome web-trending applications that give you more precise control over the actual scaling of the Y-axis, and we think they serve the tactical employees (e.g. operators and engineers) well.

The Zymergi implementation of Y-axis scaling is geared towards the time-crunched strategic users like managers and directors.

See also:

• Securing ZOOMS
• Introducing ZOOMS
• What is Metadata Search for Time-Series Data? 

Securing the Process Data Historian Search Engine: ZOOMS

by Oliver Yu

When I worked at an FDA-regulated commercial biologics manufacturing facility, access to data was limited to authorized individuals; 21 CFR Part 11 (10d) specifically calls for this control so that the authenticity, integrity, and, when appropriate, the confidentiality of electronic records are assured.

Needless to say, those who operate under cGMP regulations are worried about compliance with Part 11 have concerns with software that is trying to democratize the data.

Fair enough.


It turns out that you can have cake and eat it too.  That is, you can have an easy-to-use, web-based interface for your trend data AND you can limit the access to authorized individuals.

Within Internet Information Services (IIS), the web-server on which ZOOMS runs, the default setting is to enable Anonymous Login and to disable everything else.

The way to secure ZOOMS is go to click on the website under which ZOOMS is installed and select the Authentication Feature:

When you double-click on IIS Authentication, you get a setting for how you want this web-server to be secured.  There are 4 options to enable/disable.  The way to ensure unauthorized access is not granted is to set Anonymous Authentication Status to Disabled.

And if you happen to be running Windows Active Directory and you want to use that as the method to control access, set Windows Authentication Status to Enabled.



Assuming that you're in an environment with Active Directory, you can use Active Directory credentials to control access to ZOOMS.

Now when I attempt to access ZOOMS, here's what I get:

(I'm on a Mac using Safari to access a Windows IIS server running ZOOMS)

And only when I input valid Windows credentials am I granted access to ZOOMS.  At this point, access to ZOOMS has been limited from everyone with network access to everyone with a valid domain account.

If you want further restrictions, you simply need to set up ASP.NET URL Authorization where you specify the exact role that you want to allow access.

If you want to create an Active Directory group for just your users, you can do so.  If you want to grant access to an existing Active Directory group, you can do so.

The key here is to Allow first, Deny last.

The point in all this is that you don't have to sacrifice the freedom of your information for the sake of security.

For more questions, contact Zymergi Technical Support at 650-646-4996.

See also:

• ZOOMS: Semantic Search Engine for Time-Series Data
• ZOOMS = Metadata Search

Semantic Search Engine for Time-Series Data

It's the year 2014 and there are several updates to the Zymergi Object-Oriented Metadata Search (i.e. a semantic search engine for time-series data).

The first update is that we've moved to HTML5.  This means that we're dropping support for for IE8 or less.  We're proven on the latest versions major web-browsers: Google Chrome, Mozilla Firefox and Apple Safari.

Here's a screenshot of what the search page looks like.  There's no equivalent of "Google Instant" but given the metadata inputted by the administrator into ZOOMS, the software is able to guess what words you are meaning to type:

The second update is that ZOOMS uses a CSS Framework called, "Bootstrap."  Bootstrap was developed by Twitter and they gave it out for free.  The deal with CSS Frameworks is that someone has already done a lot of the heavy web-page styling.  You just need to customize it a bit to fit your look.

The other thing about Bootstrap is that it is "responsive," meaning the webpage will "respond" differently depending on the size of your screen.  Responsive web design (RWD) refers to whether or not a website is optimized for mobile devices.

No, we aren't wading into the time-series-data on mobile devices.  We actually believe that Transpara has a great product in Visual KPI.   We actually don't expect customers to be using ZOOMS on a phone or tablet on a regular basis, but we acknowledge that it can't be helped; in those cases, you can get your trend data with ZOOMS.  Besides, it came for free, so why not add this value?

The last update is that the entire product has been refactored.  The changes are all under the hood, but we've applied the MVC design pattern to separate the functions for a faster, more technically supportable system.

Below, you see a search engine results page (i.e. SERP) for the search terms inputted above.  ZOOMS tells you exactly which typed words has which meaning:

• Reactor1 was identified as a unit.
• Temperature was identified as a parameter.
• Volume was identified as a parameter.

ZOOMS is figuring out what the typed words mean and combining it into a search result.  In this case, someone looking for the reactor temperature and volume may be interested in a trend with each of these two traces plotted.

It is by far the fastest way for an untrained individual to get/view time-series data in a process environment.

We'll be disclosing shortly, but in addition to existing features, there are exciting updates in the coming months… like the ability to run without connecting to other data historians (yep, there's going to be an internal one).

Want a test drive?

The Zymergi Mission

Zymergi sees a world where there is greater wealth than today: a world where individuals have more time to spend on what they want (rather than what they must to do).

Look around the house. Chances are, there's a clothes washer/dryer. The washer is a reactor/centrifuge combination; the dryer is a tumbler... chemical processing units. Now, you don't have to waste time at the river with a washboard washing your clothes.

Oh, there's a fridge and freezer? That's refrigeration process with replete with compressor, expansion valve and working fluid and now you can store foods longer and not have to salt your foods or waste time on extra trips to the grocery store.

Gotta a car, too? That's the chemical processing unit where you convert hydrocarbon to CO2 + water, heat to work to get from point A to B. Now you can get to that job 40 miles away in 40 minutes instead of 8 hours.

The time you get back is possible because of these chemical processes. And these chemical processes are executed by these machines (i.e. "units"). And when you look beyond the symbols of wealth (which happen to be fancier units like yachts and airplanes and sports cars), you realize that the wealthy have limitless time for leisure, while the poor do not. No truer words were spoken when they invented the phrase, "Time is money."

Zymergi thinks the path to creating more wealth for us, our customers, and their customers is to help people who run chemical processes better "see" what their units are doing and get more out of them. The easier it is to "get data," the more time our customers get to "understand the process." And the more process understanding, the greater ability to change the course of the process for greater profitability.

The world where machines do more work means a world where humans have more leisure. Automation is the answer.

Even the world of Star Trek, where does this time...

...to explore strange new worlds, to seek out new life and new civilizations, to boldly go where no man has gone before

come from?

They (the crew) are enabled by the Starship Enterprise. The starship is simply a plant that manufactures light-years. What runs the enterprise? A "computer" that is simply the starship's SCADA. Again: automation systems.

In virtually every scenario of the future we want to live in, automation... units... chemical processes are there enabling humans to have more time... more leisure.   This future of greater time and wealth is the Zymergi Vision.

The Zymergi Mission: the relentless pursuit of excellence in systems integration and automation software is the straight-line that you can draw from our reality today to our vision of tomorrow.

  

Ron Taticek, In Memoriam

UPDATE: Memorial Service to celebrate Ron's Life will be held at 3pm on Saturday, May 3rd (2014) at:

Grissom's Funeral Home - map
267 East Lewelling Blvd.
San Lorenzo, CA 94580

---
I learned today that my mentor and long time advocate, Ron Taticek, passed away. Most recently, Ron was the Director of Technology for Genentech's Vacaville Operations.

As I look back on my career, he's the guy in the last row of the bleachers rooting for me the entire way.

In 1994, Ron was completing his doctorate when I was a freshman at Cornell; our paths didn't cross until 1998 when I arrived in California - jobless and he was a Sr. Group Leader in Cell Culture Fermentation R&D (CCFR&D) at Genentech.

I knew to call Ron because our advisor, Mike Shuler, told me, "If you're headed west, look up this guy, Ron Taticek and he'll help you."

I was skeptical, but just as Shuler said, Ron didn't hesitate to help out a kid he'd never met.... whose only connection was through his Ph.D advisor. I called poor Ron once a week asking about jobs at Genentech. (Back then, I thought the tenacity = employment); I called him so many times that I know his phone number to this day: 650-ABLE-JOY.

He agreed to have an "informal interview" where he had me come to South City and he showed me around and left me with the words, "Companies like Genentech value professional scientific experience. So go and get at least a year's worth of professional scientific experience and then give me a call."

It turns out that in August 1998, I got a job and it was an awesome medical device job; the only thing wrong with it was that it wasn't bioprocess. And so in August 1999...with a year and two hours of professional scientific experience under my belt, I dialed ABLE-JOY.

Ron answered cheerfully (he had a very smooth phone voice) and said, "Great to hear from you, there's a junior engineer position up at our new plant in Vacaville: I'll forward your resume!"

I got that Associate Engineer position in October of 1999 at Vacaville and in 2003, I had the opportunity to work directly with Ron when they were transferring Avastin to Vacaville.  It was because of Ron's career jumpstart that I had the opportunity to work with the cell culture greats like Jess Bergevin, Dan Stark, Bob Kiss and Cartikeya Reddy (just to name a few).

What a lot of people don't know is that Ron Taticek was the lead cell culture engineer in developing the Avastin (version 1.0) cell culture process. The reason - in part - why literally tons of bevacizumab have made it to market is because of Ron's work developing the cell culture process.

In October 2004 when Ron became the Associate Director of Ferm(entation) MSAT, he asked me to join him in South City. I accepted and reported through him until I left Genentech in 2007, creating awesome tools, doing awesome support, and working with universally awesome people.

Since 2007, I'd check in with Ron a few times a year; we even sat down to have salads (apparently, our respective wives had put us on diets).  It was then that I realized I've known him for 15-years. And it was then that I realized how "there" he was for me since I was some fresh-out-of-school kid.

Jan 2013 was last time I would see Ron in person... and I'm glad I got the chance to tell him how much I appreciated him. We parted ways like usual: promising to keep in touch as he ambled off to another meeting.

If there was anything left for me to say to you, Ron, it would be:

Your life's work mattered to millions of people. Your mentorship and advocacy profoundly impacted my life.  And your true legacy: N & N will miss you dearly. Though you left us too soon... and it's not fair... you rocked this life by making a difference.

Rest in peace.

Lowest Number of FDA Inspections Since 2008

FDAzilla has reported that the FDA conducted 39,774 inspections for the year ending 2013.

I guess on any given day, there are at least 109 auditing teams running around the world inspecting our GMP-regulated existence.

But it turns out that 39,774 inspections is a 5-year low for the FDA.  As FDAzilla puts it, this is the first time since 2008 that the FDA has dipped under 40,000 inspections per year.

Was it the government shutdown?  Was it the sequester?  Who knows?

As we ramp into prime inspection season (Q1/Q2 are known times when there's a lot of FDA activity), how are you guarding against getting dinged by the FDA?

• Inspect the (FDA) Inspector
• FDAzilla offer for Zymergi blog readers
• Buy 483 documents

ZOOMS 2 - Fastest Way to View Trends

So in addition to biologics manufacturing commentary, it turns out that Zymergi is actually a for-profit business that provides software, consulting and technical support.

For 2014, we're pretty much booked on the technical support side, but I wanted to take some time to talk about our software products.

Our flagship product is ZOOMS, which is an acronym for Zymergi Object-Oriented Metadata Search.  And in 2014 - thanks to Edward Snowden, more people know what metadata is than in 2008 when we started ZOOMS (v1.0)

So why are we interested in searching metadata? Well, let's take a step back. When working the front lines of campaign monitoring and process support, we noticed that viewing trend data (i.e. tag values plotted against time) was the principal component of process understanding. The more trends a person reviewed, the more process knowledge they gained and the more they understood normal from abnormal.

And in all that time, very few people actually learn to speak "Automationese."

"Hey, did you see that weird thing going on with T100.GLOBAL.CLX.AI05.PV?"
- No one ever

In the automation world, everything is a tag. In the Manufacturing Sciences world, everything is about a measurable parameter within the process. So when you listen to the process scientists and engineers talk, it's always about some parameter (e.g. "Optical Density") in some unit (e.g. "Tank 100"). That right there is the metadata of a tag.

The tag takes care of the Y-axis on the trend. What about the X-axis?

The X-axis deals with time-windows: start times and end times and the metadata around the time-windows are called, "batches." Specifically using S88 terminology, people involved with campaign support are interested in Unit Procedures, a.k.a. "unitbatches."

I'll leave the formal definition of "unit procedure" up to the automation gurus, but to a Manufacturing Sciences data engineer, a unit procedure is a process that happens on a piece of process equipment.

So say you're making rituximab in production bioreactor T100 using the version 1.1 process and batch R2D201 ran from 20-Dec-2013 to 28-Dec-2013... that there is a unit procedure:

batchid	unit	product	procedure	start time	end time
R2D201	T100	rituximab	production culture version 1.1	20-Dec-13	28-Dec-13

The metadata around this time-window (i.e 12/20/2013 to 12/28/2013) are as follows:

• R2D201
• T100
• rituximab
• production culture version 1.1

So it stands to reason that if an internet user who knows nothing about a subject can type keywords into Google and get the most relevant results on that subject; that in 2014, a process citizen who doesn't know too much about the process ought to be able to type some keywords into a webpage and get some process trends.

And now they can: Introducing ZOOMS 2:



Learn More

Yes, We're Still Alive

Yup, we're still alive around here.  Crazy busy with some exciting things afoot.

Next week, we're at the OSIsoft PI User's Conference 2014 in San Francisco:

So if you're around and want to meet up, come find me @zymergi.

ZST is 67% faster.

For those of you who don't know, the ZST (Zymergi SQL Tool) is a Windows-app that gets your database information and puts it in any format you want.  Typically, customers want to convert their relational data to HTML5 so that they can view (and act on ) it with a desktop or mobile-device browser.



Well, there's good news.  ZST is now 67% faster.  Now, this increased speed may not matter in an enterprise environment where apps are feature-rich, but slow.  But for our customers with external-facing websites, more speed = more revenue.

So why would the enterprise customer want ZST?

Well, the first thing is that this is commercial off-the-shelf (COTS) software.  Secondly, you're essentially adding search and browsing capabilities to your existing relational data.  And ultimately, you're looking to institutionalize tribal knowledge.

• Boosting speed of ZST
• ZST Support Page

Zymergi SQL Tool (ZST): Could not load file or assembly 'System.Data.SQLite'

For ZST customers moving to 64-bit environments, you may run into the:

Could not load file or assembly System.Data.SQLite error.

This error is caused by the default setting of IIS7 to not use mixed-mode assembly. What is mixed-mode assembly? Well, when you have 3rd-party software libraries written in on x86 systems, you're in a mixed-mode. Since this is disabled by default, when you migrate ZST to IIS7, you're going to get this error.

Here's the solution:

Step 1: Open IIS 7 Start > Run > Type in 'inetmgr'.

Step 2: Select Application Pools.

Step 3: Right-click Advanced Settings...

Step 4: Under General, set Enable 32-Bit Applications to True

Once get this going, you ought to be back in business.

Thanks to StackOverflow for this solution.

Multivariate Analysis: Pick Actionable Factors Redux

When performing multivariate analysis, say multiple linear regression, there's typically an objective (like "higher yields" or "troubleshoot campaign titers"). And there's typically a finite set of parameters that are within control of the production group (a.k.a. operators/supervisors/front-line managers).

This finite parameter set is what I call, "actionable factors," or "process knobs." For biologics manufacturing, parameters like

• Inoculation density
• pH/temperature setpoint
• Timing of shifts
• Timing of feeds
• Everything your process flow diagram says is important

are actionable factors.

Examples of non-actionable parameters include:

• Peak cell density
• Peak lactate concentration
• Final ammonium
• etc.

In essence, non-actionable parameters are generally measured and cannot be changed during the course of the process.

Why does this matter to multivariate analysis? I pick on this one study I saw where someone built a model against a commercial CHO process and proved that final NH4+ levels inversely correlates with final titer.



What are we to do now?  Reach into the bioreactor with our ammonium-sponge and sop up the extra NH4+ ion?

With the output of this model, I can do absolutely nothing to fix the lagging production campaign. Since NH4+ is evolved as a byproduct of glutamine metabolism, this curious finding may lead you down the path of further examining CHO metabolism and perhaps some media experiments, but there's no immediate action nor medium-term action I can take.

On the other hand, had I discovered that initial cell density of the culture correlates with capacity-based volumetric productivity, I could radio into either the seed train group or scheduling and make higher inoc densities happen.

by Oliver Yu

Contamination Time Window Redux

by Oliver Yu

There's a little LinkedIn brouhaha going on regarding the calculation of contamination time windows.

In the perfect world, there are no bioreactor or fermentor contaminations.

But if you were to have contaminations, the next best thing would to omnisciently know exactly what the true root cause is.

Since omniscience is not an option here, the next best thing is to narrow down the list of sterile-envelope manipulating actions on the contaminated bioreactor and perform root cause analysis to come up with MPC (most probable cause).

There are a lot of parallels between a crime scene investigation and bioreactor decontamination response. Just as crime-scene investigators attempt to determine the time of death to rule out potential causes outside the time frame, it is a good idea to compute a contamination time-window to rule in/out potential causes.

There are several assumptions in the contamination time-window calculation and which assumptions you use depends on your organization's risk profile. As in, which is worse?

• Coming up with too narrow a time window and risk eliminating the true root cause
• Coming up with too wide a time window and having too many potential causes to investigate?

To summarize, you run the exponential growth equation twice: the first use is to try to figure out the growth rate of the organism.

Step 1: Collect sterility samples and send samples to QC Micro.

Step 2: Get 2 pieces of information from QC Micro:

1. Concentration of contaminants in last sample (X)
2. Timestamp of last clean sample. (t₀)

Note, you already know the timestamp of the last sample (t).

Step 3: Pick an X₀ and try to compute μ.

X = X₀ e^{μ(t - t₀)}

bold is known.
blue is estimated.
red is what you're trying to determine.

The rational number to input here is the limit of detection of your sampling method.   If you're going for a narrower window, pick a higher number.

Now you have an estimated growth rate (μ) of the contaminant.

Step 4: Now try to determine the earliest insult (t₀)

X = X₀ e^{μ(t - t₀)}

Again, bold is known; blue is estimated; and red is what you're trying to determine.  Here, the key is to guess the concentration of the initial insult.  If you're afraid of ruling out the true root cause, pick 1 CFU.  If you're going after a realistic time frame to reduce scope, pick what you think a small bolus of contaminants would contain.

When you've solved for t₀, the second time, you now have computed the earliest time of contamination.

As you can see from this process, there are some serious assumptions.  But if applied correctly, you have a scientific and defensible basis for narrowing down potential bioreactor contamination causes and focusing your limited resources on rooting out the most probable cause.

Get Expert Contamination Consulting