Tag Archives: availability

Porting an Enterprise App to System z – my experience. Part 1 of 4: The Basics

February 13, 2018 Jim Porell 4 Comments

At the end of 2016 and lasting a few months into 2017, I completed a proof of concept port of a large Enterprise Application that had been running on the Amazon Web Service Cloud to Linux for System z. This was a Docker based application written in Java…so of course, it would be trivial to port. WRONG. While the application is in Java, it called many pieces of open source code. Much of that code hadn’t been ported to System z yet or wasn’t widely adopted. What I thought was a very simple exercise turned into a six month effort.

What I’d like to do, via a series of blog entries, is share my experience in the hope this might help some other organization decide to do a similar porting task. While I’ve been working with mainframes for decades, this was my first Linux porting experience. So I’ll be describing how this experience helped me to Master the Mainframe, though that title seems reserved for university students.

This could be a book, but by breaking it up, it might be easier to understand.

The Basics: High level overview of the application, the development environment, the system set up required to begin the porting exercise and the scope of the port.
The Good: The people who assisted and taught me, the things that ported easily and the simplicity of getting started via the Linux Community Developers system.
The Bad: the new open source for System z, the modifications necessary to open source to run on z, the debug experience and the time necessary to complete the porting process.
The Future and Value. Regardless of the bad experience, there is a great business value in getting these types of Enterprise Apps on System z.

The Basics

This entry is more about the basic desktop development environment and targeted production on x86 based cloud servers. This is the traditional development environment and primary target of the applications. I needed to fit in and work with this environment before I could ever consider doing the unique activities necessary for success on Linux for System z.

Application Overview

Because of the proprietary nature of the application and intellectual property, I’m not going to name the vendor or application. This overview of the workflow is simplistic, at best, so as to not give away any trade secrets. The vendor is an early start up with an application to handle biometric authentication in a marvelous way. This application has a callable interface to start a request and then, using cloud based services, does some communication with the end-user, does some analytics based on a number of system defined characteristics, logs a number of things for diagnostics, audits and future analytics, provides a go/no-go decision back to the original caller and has a number of applications and user interface applications to manage the cloud deployment. Finally, they have an enormous test suite to emulate and automate the entire end to end workflow.

Development Environment

This vendor was doing all of their development for the x86 platform and originally with any Linux version supported by Amazon Web Services. This included Centos/Red Hat versions. Their first development environment used Maven tooling and pom.xml scripts that targeted deployment into Docker containers. They used Github capabilities to clone and manage the source code libraries within their business.

The first major effort was for me to establish a development environment on my computer and prove that I could work with and build a workable x86 version of the code. My computer of choice was a MacBook Pro 2010 model running the latest MacOS at the time. First thing to do was turn my MacOS into a real developers machine. I installed xcode, Atom, SourceTree, Filezilla and Docker which enabled me to look like a Linux system, edit source files intelligently, manage access to the source files, facilitate cloning of source and execute the code. There were other local variant software that I needed to install using a script that was provided to me. I love the Mac, as did the vendor, who’s entire team used it, so that was really helpful. I then needed a VPN into their system and I was off and running. I used this set up for about two months. One thing I learned, painfully, that the 2010 Mac was SLOOOOOWWWW. What would take 15 minutes to do for them might take me over an hour. So I decided to upgrade to the MacBook Pro Touch Bar quad-core 16GB memory laptop. Now my work completed faster than their 15 minutes, which was a blessing. I can’t stress enough the value of a good starting point on the desktop or laptop for this type of development! It was life changing to me.

Open Source and Operating System Dependencies

The first version of the vendor code used Centos/Red Hat as the target deployment environment. This code runs over 50 Docker containers. Each container is intended to be as small, memory wise, as possible, so it is scalable in a largely virtualized environment. As mentioned earlier, they also used Maven and pom.xml scripts to do their container builds. Each container had a script that would gather necessary pre-requisite open source parts, their Java code and then do the build so there was an executable container. Naming conventions, versioning and more were part of these Maven scripts. 90% of the open source code used was available in a binary form as either an RPG, ZIP or TAR file. Those binaries were either copied into the vendor’s library system or accessed via a URL and dynamically downloaded from the internet during the build process. I’ll get into the System z ramifications of this in the Good and Bad blog entries.

This is the development environment I began my first phase of the port. The prototype I was building was only for a functional test to prove the code could work. We intended to accomplish our test goal with only 40 of the 50 containers being ported. We completed what we thought was a good test level of code after a few weeks of my porting. But then we identified some critical test containers were missing. Unfortunately, the vendor didn’t use the same library management rigor for their test suite and I was going to have to re-base my code.

Rebasing the code and changing Development environments

Unfortunately, that was the tip of the iceberg in changes. I mentioned this was a startup vendor. They had two very large customers that were testing the code when I started. They realized they had a scaling problem, early on. They also realized they had some development inefficiencies. When you get a RedHat, SUSE or Ubuntu distribution, there is a lot of software in the package, like getting the z/OS operating system, MacOS or Windows. As such, the kernel of the large distribution Linux systems can start at 250MB and easily be over 750 MB’s. When you add 100’s of virtualized containers, each having that size as the basic footprint, the overall system runs out of memory pretty quickly. However, if the kernel can start at 18MB and run about 50MB, then greater scale is possible. As development of this application began, the Alpine Linux distribution began and it met the small size requirement. The vendor began to rebase all of their test code and as much of the open source code as they could on Alpine to take advantage of this reduced memory benefit. That was and is an excellent business decision on their part.

Maven is a fairly complex environment for building docker containers. It works. Both the vendor and I proved that it could work. However, in addition to open source code, there are now open docker containers that can be leveraged, as is, to be included in place of an open source binary. However, in order to do that with Maven, the Docker definition files of these open containers must be cut and paste and then modified as part of the Maven script syntax. And each time the container definition changes in the open source world, the Maven scripts need to be hand modified. So the vendor dropped Maven as the base for their container build environment and switched to using Docker build definitions directly. Again, I applaud the vendor for doing this. It simplified the development environment, it gave them access to additional open source code repositories and made everything easier to manage.

The unintended consequences of the vendor’s change from Maven to pure Docker and Centos/RedHat to Alpine was I had to start all over on the port. I’m going to save the details of that for the Good and Bad statements as they are directly applicable to System z.

As far as Linux for x86 cloud environments, this vendor has a world-class development environment, working to create the most reliable, secure and efficient application possible. Ultimately, those attributes must apply to System z deployment as well. I’ll be covering that status in the other blog entires.

Hybrid Computing

Porting an Enterprise App to System z – my experience. Part 2 of 4: The Good

February 13, 2018 Jim Porell Leave a comment

I provided a simplistic overview of what I intended to port to Linux for System z in Part 1. The original application was built for x86 systems. As such, all binaries are built to run on x86 systems. The Docker containers that these applications run in are x86 binaries as well. So my job was to create the Linux for System z (aka S390X) binaries, with as little change as possible.

I also mentioned that this was a start up vendor with whom I was working. I had done some business work to show them the value of porting the application to System z, but they were neither skilled in, nor able to afford their own System z. So I gave them the challenge to let me prove to them this could be successful and they took me up on it and agreed to work with me.

Vendor Development Team

While a small development organization, they still had over 25 very proficient programmers and testers. I was extremely fortunate to have their lead developer as my mentor. He and I would meet at the same time, for an hour every day to check on progress, educate me or diagnose any problems I might have so that I could make progress for the next day. Most important is he was learning about the mainframe and intrigued by the possibility of business success as I was, so it was a great experience for both of us. I greatly appreciate the time and effort he put in to make this a success.

Linux Community Development System for z

Where do you find a mainframe? You ask the Community Development team. Eva Yan at IBM was instrumental in approving the vendor and I to get access to Docker containers on the mainframe. Cindy Lee at IBM was fantastic, with her team, to help show me where all the open source for z was available in the community and Martha McConaghy at Marist College, the host for vendor access to the LCDS was terrific in helping me to keep the system running.

Docker is a great place to work with portable code. My development environment was an x86 Docker container environment that pointed to the S390X Docker on the LCDS system as the target deployment environment. I’m not going to spend time giving you the details on the set up, but suffice to say it all works well.

Scalable Virtualization

I didn’t mention before that the vendor is on a different continent. So imagine from my laptop, a VPN to the vendor’s libraries where some code is downloaded, merged with code on my desktop, Docker on my desktop puts all the parts together, ships it securely to the Docker on the mainframe image, does the build and sends results back to me. So if this process took 10-15 minutes to do on my laptop, suffice to say, when you add up the networks and bulk distribution of code between systems and do the build, it’s going to take more time than a single system. Doing a single container build, for the first time, was never correct. My mantra, for years, has been “Next time for sure!”. I’d fix what needed fixing, get a little farther the next time, repeat the mantra and try again, until finally, I’d get a successful build. The time or performance isn’t a problem when building a single container. It’s when you build 40-50 containers at once, or as I liked to call it “The Big Bang”. Then it was hours to do the build on the mainframe, instead of an hour on x86. You’d think that was the bad, right? It was good, because a call to Eva, requesting some more memory and processors and I moved to a very competitive deployment environment. For just like my MacBook 2010, which was under configured for this scale of development, the initial Linux system I was given was an under configured virtual machine. With a simple config change, within moments of my request, and literally no down time, I was up on a larger Linux image, due to the magic and wonders of the underlying scalable z/VM server image.

Open Source Access

The LCDS virtual images came with RedHat kernel as the base, with some optional software included, but that was all. I need several dozen pieces of open source software to add to my environment to build my S390X binaries. Again, I don’t want to spend the money to buy a supported Linux distro for this Proof of Concept. I’m directed to Sine Nomine Associates, and in particular to Neale Ferguson. He could not have been a better ally in this effort. First and foremost, he pointed to libraries on their servers where I could retrieve many of the binaries that were necessary. It was such a relief to find many of the rpm’s I needed on their website. As mentioned earlier, I was a newbie to this kind of porting. He spent considerable time mentoring me on both basic Linux and System z specifics to keep me moving along. As important, Neale was on the Docker band wagon. He’d begun building docker containers with specific functionality. I was able to take several of his containers and imbed them into the containers I was building to simplify my deployment.

The Linux Community also has Github repositories of System z ready open source code. I bookmarked those pages and visited them often. I’m pointing links in a Bibliography in Part 4.

The real dilemma came when the vendor switched from Centos to Alpine as the base Linux kernel. Alpine was so new in late 2016, early 2017. While both are Linux derivatives, the syntax of packaging applications is different. As such, Docker builds for Centos are different from Alpine. Because I was doing a proof of concept, it really didn’t matter whether I used Centos or Alpine. However, the longer my porting took, the faster the vendor was converting their code to Alpine, so now, I would have to make “throw away” changes to support Centos.

Worse than that, there was only one person even trying Alpine on the mainframe and that was “some college kid” as a research project. How could I build an enterprise application on a system that one unpaid person was supporting? That person was Tuan Hoang and I am indebted to him. He was a Marist College student. I began contacting him late in 2016. While he had the kernel ported, there were very few packages for Alpine ported to S390X. He was quickly up to the task. I gave him a list of high priority packages. Each night, I’d get an update of what he completed. Each day, I’d build some more containers off his evening’s work. It got to the point that only third-party open source packages were not done by him. This really got my development effort going. But the best news of all was at the end of my project. Tuan had worked so hard to get his “prototype” of Alpine for System z going that the Alpine community accepted S390X as a primary target platform. All Alpine packages would be available on S390X, simultaneously to their deployment on other hardware architectures. It was painful, but it was wonderful at the same time.

Good people make life easier

What I found throughout this porting effort is there is a wonderful community of people dedicated to the support and value of System z. They were very accommodating and helped reduce my efforts greatly.

Hybrid Computing

Porting an Enterprise App to System z – my experience. Part 3 of 4: The Bad

February 13, 2018 Jim Porell Leave a comment

As I’ve explained in Part 1 Basics and Part 2 Good, I did a proof of concept port of an Enterprise Application from Amazon Web Services on x86 to Linux on System z in 2017. The good news was I got to the point I needed to, the bad news was it was more than difficult to get there.

Linux is not Linux

Open Source is open source…available to anyone. The story goes that Linux is Linux. Close, but not quite. Unfortunately, architectural chip bits (Big Endian vs Little Endian) is one of many differences and there is code that needs to change to handle these differences. There are also supported platforms, “tolerated platforms” and unsupported platforms. This is the problem with Linux on System z. The marketing hype is that all of Linux is supported on z. The reality is somewhat different. Not necessarily insurmountable, but you better know what you are getting into.

Supported Platforms

When Linux on z is a supported platform, then the packages for System z are supported in binary format, such as an RPM file for Centos/RedHat or an APK file for Alpine. This is the best case and makes development of S390X on par with other platforms like x86 and ARM.

Tolerated Platform

In this case, the code may work on S390X, but it’s a source code build. You can find instructions on Github for S390X as to how to modify the code to get it to work on the platform. But if you want to use that code, it could take a long time to

Do all the things necessary to manually modify the code
execute the code to create a binary.

Let me use an example. Couchbase is the non-SQL database preferred by the vendor I worked with. Someone within IBM is maintaining a script on Github to help others leverage a particular release of Couchbase. Since Couchbase is constantly coming up with new versions, those edits need to be constantly updated. I would have preferred a binary version of the code, but IBM doesn’t do binaries…They only do source. And in order to make Couchbase work, there are pre-requisite source modifications necessary to Go, Python, cmake, Erlang, flatbuffers, ICU, jemalloc, and v8 javascript. Manually doing all that is necessary takes a few hours. I was fortunate to take all of these changes and build a docker script that was several hundred lines long to automate the build of Couchbase by doing all this work. When I ran this container build, it took over an hour to complete. I had to do this many times before I got the automation script to work properly. And that automation is only good until the next release comes out. In comparison, with an x86 rpm, this takes a couple of minutes and the Docker script is about 15 lines long. In the end, I got what I needed, but the level of effort to get there was tremendous. I also mentioned container memory size in Part 1. This Couchbase container on z was over 1 GB memory. This put a tremendous strain on Docker and we found a few bugs as a result. The size was a combination of Couchbase and all the prerequisite code to build Couchbase. So I had to modify the Docker build to delete all the prerequisite code which included source, binaries and documentation. This got the container down to a more reasonable execution size.

BTW, when I complained to IBM leadership about the lack of support for Couchbase, they suggested I use a different, easier product that was available on z. Since I was porting and not a true developer, this was not a possibility for me. I had begun negotiations with Couchbase toward this goal, but stopped working on it when the prototype ended.

Unsupported Platforms

There were two cases where neither the open source community nor the Linux on z community had guidance on how to get a particular open source program on the mainframe. In those two cases, I was able to get through the code, successfully and get a binary for System z. The good news was it was pretty simple to do. I was quite fortunate. If it hadn’t been easy, this could have ended the project earlier than I had hoped.

Docker containers are not portable across hardware architectures

I’ve seen some hype that once you get it in Docker, it’s portable to any Docker. I’ve heard a few mainframe customers believe any Docker container can run on System z. I’ve also seen articles in IBM sponsored magazines that purport this to be true. This is a combination of marketing hype and misunderstanding. It all depends on the container architecture/binary and source code. Typically, a container binary for a particular architecture, such as x86, should run in a Docker container on any x86 platform, even if it’s a different operating system running Docker. For example, Docker running on x86 version of RedHat 7.3 could be running containers with RedHat, SUSE, Alpine, Ubuntu, etc, as long as they were built for x86. Similarly, I ran Docker on a RedHat 7.3 image for Linux on System z, and had containers with Centos and Alpine running with binaries for S390X.

The only containers with source code that were portable were built exclusively with interpretive languages, such as Java or Python. Those could be portable across hardware architectures. Many of the test cases used by this vendor fit into that category. However, as soon as one of those interpretative languages makes a call to open source code middleware (e.g. Couchbase), then the container is no longer portable across architectures because the middleware is not supported across architectures.

Docker Stability

When I started this project, Docker on z was pretty new. Once in a while, it would have issues. Only a couple of times did it require Marist College to restart my z/VM guest. The other times, it would automatically recycle itself and get running again. I believe it’s improved since we began the port effort until now, but it’s been a few months since I tried it. I’ve heard from others, though, that the experience is better. During our Big Bang builds, we would peg each of our System z processors at 100% busy for a few hours. The fact that it would stay up and continue processing is a testament to the reliability of those large code tests.

Ultimately, I have a wishlist for the Open Source Community on z:

Where source code changes are necessary, such as with Couchbase described earlier, supply a Docker build file to automate it for anyone that wants to do the build. It would be so much faster.
Continue to lobby third-party open source middleware providers to support system z. In many cases, it takes a vendor, such as I was working with, to create that business case jointly to get it done, but doing that will lead to more usage on the platform. If you build it, they will come.
Create more binary packages instead of source code update files. It greatly reduces the development time necessary for z unique porting. The more extra work necessary to support z, the less likely the x86 people will move there.

The net of all this bad is the initial effort to support the mainframe is longer than it would be on x86. However, if you have the patience to get to Part 4: The Future and Value, you’ll find that you should be rewarded for the effort.

Hybrid Computing

Porting an Enterprise App to System z – my experience. Part 4 of 4: The Value and Future

February 13, 2018 Jim Porell 2 Comments

In Part 1 The Basics, Part 2 The Good and Part 3 The Bad, I’ve explained I did a proof of concept port of an Enterprise Application from Amazon Web Services on x86 to Linux on System z in 2017. The good news was I got to the point I needed to, the bad news was it was more than difficult to get there. But why did I go there in the first place?

The vendor for the Enterprise application was targeting the Financial Services industry for their initial deployments. This is the primary customer for IBM System z. Their beta customer is running z/OS transaction processing via CICS, but wants to authenticate customers using this vendor’s product running on Amazon Web Services. In order for CICS to call the AWS Cloud, it has to launch Websphere on z/OS to call the vendor’s service on AWS. The vendor’s application has to do it’s task of authenticating users and get all the way back to CICS in less than 18 seconds so the transaction doesn’t time out. It’s a really powerful use of the vendor’s application and valuable to both the consumer and financial institution to avoid potential fraud or cybersecurity scams.

Java and Analytics run better on z/OS

I was told this vendor wrote all their code in Java, so I immediately began a plan to get this running within z/OS, since Java runs so well there, especially on the z14 systems. I also knew that in the time allotted to run on AWS for those 18 seconds, only three biometric/analytic tests could be completed on behalf of the consumer. I hypothesized that if the vendor app ran within z/OS perhaps up to ten analytic tests could be completed using the outstanding analytics and Java performance. However, once I learned of the number of open source middleware programs required and the complexity of porting them to z/OS, I went to Linux on System z as the target port.

Linux on z as a private cloud has more value than a public cloud

Using RDMA as the memory based communication between z/OS and Linux LPARs, I know it will take a bit more time than running inside z/OS, but much less time than going to a public cloud, so I hypothesized that eight analytics tests could be done instead of the three on AWS. And regardless of z/OS or Linux on z implementation, the vendor agreed that the software price would be the same as AWS. The net is, z would have additional analytic value, and given it’s hardware and software integrity and reliability, it would offer better security and business resilience than any public cloud provider.

So that’s what I set out to prove. Sadly, I got so close and the vendor changed their mind on their business strategy. They received a significant new round of venture capital investment, signed up several new financial firms to try their code and they decided to stick to their current cloud plan and stay off the mainframe, for now.

I still believe that my hypotheses as to the performance and value were correct. But the activity ended just before I was able to prove that. However, the exercise did confirm the possibility of getting the product on the mainframe successfully.

Docker inside z/OS? That would simplify things!

But what else is possible? I said in Part 3 that Docker containers are not portable across architectures. However, they are portable within the same architecture. There are some prototypes underway for Docker to run within z/OS. Given the way Docker works on other platforms, it would infer than any Linux on z containers could run unmodified within z/OS. If Docker for z/OS were to run on a zIIP processor, there would be no software license hits for z/OS. If that all comes to pass, that could lead to significant transaction and analytic value within z/OS and greatly simplify the system management requirements for these types of hybrid workloads, while improving the overall security, resilience and performance and reducing the operational costs. I would hope that a public announcement of this capability is not too far in the future.

Savings and Operational Strengths

That, my IT friends is a win for everyone. Any of the bad associated with a slightly more complex development environment can quickly be eradicated with a greatly reduced operational expense that has greater operational benefits than any alternative architectures might try to demonstrate. This type of workload makes for a very compelling end to end benchmark comparison as well. So while I didn’t succeed in getting the enterprise application to market, that was because of a business decision rather than a technological impediment. And the business decision was tactical, based on their new financials.

I learned a lot and documented many of the short cuts I took and set up required to make this development effort possible. I’m happy to share the experience if you’d like to undertake your own development effort. While I thought the end of the project was a failure, it’s unintended consequence, with the efforts of the great Linux for z community identified in Part 2, is that this will be easier porting for everyone that follows.

Bibliography

LinuxONE and Linux on z Systems Open-source Team

LinuxONE Developers Works

Neale Ferguson’s pre-built Docker containers for z

GitHub repository to S390X open source scripts From this page, search for the package you are interested.

Hybrid Computing, Mobile Computing, Security

Closing the gap on technology evolution

January 15, 2016 Jim Porell Leave a comment

I recently saw a blog post by one of the Federal CIO’s. I can’t argue with their observations, though I think we may disagree on how to tackle the problem. That CIO is going to post their direction in future posts. I’m going to take a shot at my own direction in this post.

The following graph demonstrates that the US Government IT is falling behind Fortune 500 firms and way behind internet startups.

Federal CIO study graph

I remember having this debate with an IBM General Manager years ago when he was considering outsourcing some operating system components thinking that all programmers are created equal. There is a huge difference in maintaining a legacy of millions of lines of code vs. starting from scratch with something new. As important, starting over AND maintaining all the rules and regulations of the legacy, is also a very difficult proposition. It takes pre-existing knowledge for success.

This CIO faces a problem that is similar to many other businesses. It’s true for mainframes as it will be for Microsoft Windows and Linux systems in the future. There are millions of lines of “legacy code” in languages that are less popular today than they will be in the future. The inference is to move away from the legacy code toward a modern language where there are more skills available. As a factoid, there are more ARM chips in the market today than Intel chips. There are more applications being developed for iOS and Android than for Microsoft Windows and that’s way more than being developed for mainframes. So that might lead someone to believe that’s the programming model of this generation. And as I’ve said in an earlier post, if your IT career began in the 1990’s and you hated mainframes, you were right….at that time….

But like everything, time changes things. IBM and vendor partners have dramatically changed what the mainframe was into a more modern computing environment. IBM spends over $1B in R&D for each generation of the mainframe that comes out about every two years now. I’m going to park that, for a moment, to go to another topic, that is more relevant to the skills discussion.

Patterns

Programming is about patterns. Patterns occur at a process level, in languages and in behaviors. There are three broader patterns at work here. Systems of Record, Engagement and Insight. I’ve written about that before, but Record deals with transaction processing, Engagement deals with the end user interface and Insight is about analytics. Most programming being done today is around systems of engagement – taking advantage of enhancements in smart phone, wearable tech (e.g. watches and fitness) and other devices that are the Internet of Things. GPS, accelerometer, touch, voice and biometrics are just a few of the advances that improve the human computer interface. The mainframe has avoided this programming area completely as a native interface. That makes complete sense. Ignored by many, though, is the fact that the mainframe has fully embraced leveraging those capabilities through interoperability and standard formats and protocols. They enable hybrid programming to reach out to those interfaces to simplify the deployment of systems of Record. In addition, they’ve integrated with Systems of Insight to enable real time analytics to be applied to traditional systems of Record to reduce risk.

This link will take you to a tremendous video about the z13 server and its ability to satisfy these new capabilities. Warning – it’s 30+ minutes long.

Where will the skills come from?

Another fear raised is that schools no longer teach “mainframe”. Perish the thought. While there are fewer “mainframe” schools than teach commodity system programming, there are a wealth of schools across the world that are part of IBM’s System z Academic Initiative. Checking their website, there are three in Maryland, close to the Federal government and very close to the agency head writing the blog. But you know, “you can’t trust the marketing” materials put out by a vendor. So I went to the Loyola College of Maryland, University of Maryland Eastern Shore (UMES) and Prince George County Community College web sites to see what they said about the IBM Academic Initiative. Honestly, the info I found was from 2011-13, other than Prince George which was up to date. So I reached out to the schools. UMES responded quickly.
“First and foremost, I would like to inform you that we are actively involved in the IBM Academic Initiative. Dr. Robert Johnson is the Chair of the Department of Mathematics and Computer Science is the lead person in the initiative. Further, they are currently in the process in moving into our new $100 million Engineering and Aviation Science Building which will significantly enhance our capabilities to support the initiative.”
Here’s a brochure for their program.

Most importantly, success is not a two-way street between IBM and the schools. It’s four way, including businesses/agencies and the students. The best schools will work with businesses to provide internships with students PRIOR to graduation. There is generally a very high (close to 50%) success rate in those students choosing full-time employment at the business they did an internship. I strongly encourage any business or agency concerned about future skills deployment to reach out to these schools and work directly with them. Experience shows that you’ll be very pleased with the results. UMES gave me their cell numbers if you’d like to reach out to me for a direct introduction.

Adopt New Technologies and dump the old?

The collective wisdom of the Federal CIO’s seems to point to new technologies as the “future” of programming. The referenced blog points to Uber, Siri and Facebook as examples of such applications and suggests they may be irrelevant in five years. (See Myspace as an example). New technologies grow up in a vacuum. There is no maintenance legacy. It doesn’t mean the legacy can’t work with them, though. A prior blog entry looks at 22 emerging technologies and their relationship to the mainframe and how hybrid computing can solve new business problems.

Let’s consider one of the new, cool tech referenced: Uber. I happen to have a chauffeur’s license (a story for another time) and am very familiar and active with Livery legislation. The Uber mobile application is actually very simple and easy to recreate. What makes them successful is their business model and practices. They hire drivers as contractors, therefore no tax consequences for Uber. They avoid the bureaucracy of Livery laws.

There is a state law that enables the New York City Taxi and Livery Commission (T&LC) to regulate who and what can be operated within the boroughs. This is for the “safety and comfort of passengers”. However, it’s big money. Medallions, per cab, have cost up to $750,000 just to put a car on the street and the T&LC limits the number of medallions. Cars from outside the T&LC are not allowed to make more than one stop in the city. They cannot pick up a passenger at an airport if they dropped them off more than 24 hours ago. The T&LC have 250+ officers in unmarked vehicles that follow and intimidate non-T&LC livery vehicles in the city. I witnessed a stretch limo being impounded by the T&LC when an upstate Livery firm dropped off the passengers returning from a NYC funeral at a NYC restaurant before traveling north. The second stop was illegal. In any event, other states (CT and NJ) got upset with this bureaucracy. They lobbied and a Federal law resulted to allow reciprocal rights to other states to operate without joining the T&LC. But upstate Livery can’t participate. The NY Assembly and Senate have had to modify laws to create T&LC’s in neighboring jurisdictions to allow reciprocal rights in NYC locations. Rockland, Nassau and Westchester counties have T&LC’s now. This is the third year that Dutchess and Ulster have legislation to enable reciprocal rights up for a vote. The NY Assembly has passed their legislation, but the NY Senate hasn’t. Last year, they decided to wait on Dutchess and Ulster until they figured out how to allow Uber and Lyft to operate in NYC exempt from the T&LC bureaucracy. That legislation has now been created and will be voted on soon.

T&LC makes revenue on selling taxi medallions and collecting tax on fares. Uber & Lyft disrupt those economics. The livery vehicles pay $3000 per year for insurance. Uber/Lyft cut deals with insurance companies to lower that to $600/year to make them more competitive. The drivers must also have personal insurance on the cars when a fare isn’t present. Laws are now being enacted to allow “Transportation Network Companies” (TNC as they generically refer to Uber and Lyft) to get “fair access” to markets in NY without this bureaucracy. I’ve developed an app which will qualify the “local” livery company to operate as a TNC to reduce their costs and in turn, reduce the cost to consumers…will the government allow that? Will the Dutchess and Ulster laws pass? This is more about big money, venture capital and paid lobbyist getting to the legislative leaders, than the small livery companies trying to stay in business. We’ll see if the legislation and the bureaucracy will enable the small livery services to morph into a mini-Uber. The legislation enables the Commissioners of Insurance and Motor Vehicles to regulate the “TNC” businesses. The legislation doesn’t prescribe how that will be managed nor how much it will cost. By the way, did you notice that the legislation for Uber includes a lighted icon in the front and rear of the car to identify it? That’s as much for passenger safety as it is to make it easier for the T&LC police to pull over the cars if the legislation doesn’t pass. Not much likelihood of that, though, given the amount of money changing hands in Albany.

Long story short – Uber is more about business processes than it is about new applications.

Past Technology Evolution Examples

Going back to the graph, there is much to learn from prior experiences of the Fortune 500 and government agencies introducing new technology.

Learn from the Fortune 500 – the good:

Benefits processing: Hewitt Assoc and Fidelity continuously advance their capabilities. They provide integration with employer payroll systems. They have up to the minute accuracy of consumer records. They provide immediate access to Accruals and eligibility. They’ve adopted web and mobile technologies as Systems of Engagement, including biometric security authentication.

Claims processing: Travelers Insurance has historically reduced IT and people expense 10% annually while improving response times. Claims agents leverage mobile technology for accidents and disasters as input to “legacy” systems.

Learn from the government – the good:

The FBI and VA leverage mainframe virtualization to avoid IT costs of millions of dollars over commodity systems, while improving security, resilience and service level agreements. They run the same code in a different container with a superior operations model and lower costs.

All of the above use Hybrid technology which includes the mainframe.

Learn from the government – the bad:

Marine Corps – hosted by an IT supplier that gouges them on mainframe costs – three times the amount if they hosted it themselves. The IT supplier takes floor space, energy and cooling costs for an entire data center and only bills to the mainframe users. The IT group claims: Commodity systems wouldn’t be affordable if they were “taxed” with those costs. That’s why understanding the Total Cost of Ownership is a critical success factor when considering mainframe vs. commodity system costs. Unfortunately, regulations are in place that mandate that the Marine Corps use that particular IT Supplier. Other groups have bucked that policy to save money.

US Postal Service was not competitive with package tracking vs UPS and FedEx. They realized they needed to add new applications and wanted modern programming to do it. It included new engagement systems at the delivery vehicles via mobile technology. ….that’s the good. The bad – they spent $100’s of millions on redundant “commodity” IT infrastructure and copied key data and applications from the mainframe in order to host the new applications, while leaving the mainframe running. Testing and benchmarking have demonstrated that adding the new applications to the existing mainframes would have avoided millions in costs and operations complexity, while simplifying the architecture and improving SLA’s. With package shipping volumes increasing annually, they’ve continued to upgrade the mainframe each year. They are just spending too much overall. While they collaborate between the systems by moving data, they could save more if they shared the data in real-time.

Prescription for change

While a prescription for change is forth coming in the CIO’s future blogs, let’s hypothesize some changes for their benefit.

Modernization of the development environment

Rational tools – They move the mainframe application development to commodity systems. This moves 80% of the development off the mainframe to reduce IT costs. They provide tools to modernize and document the “legacy” applications and simplify their maintenance. They provide seamless test to the mainframe and other platforms of deployment choice. One large business has 1000 Java developers for commodity systems, 400 Cobol programmers for the mainframe and 50 developers familiar with Java and Cobol to enable hybrid programming and integration. All use the same Rational development front end. From a skills perspective, the mainframe development can now look and feel exactly the same as development on commodity systems. This eases the skills and knowledge requirements to start.

Language modernization:

Cobol Copybooks – the means to define data structures – are now sharable with web services and those services can launch from Cobol. More on that in a moment.

Chip Speed

The System z13 server runs dual core 5GHz processors. Benchmarks show that Java runs faster here than any other platform. The video referenced earlier provides specifics. With direct access to databases and files, business applications can have better performance than other architectures. With fault tolerance and an improved hardware and software security architecture, the result is a very price competitive hosting environment for new workloads.

Risk and Fraud analytics

Financial services businesses are doing real-time analytics in the middle of their System of Record transaction programs to assess risk and avoid fraud. Leveraging the Copybook capability, they can call out to leverage the 1000+ processors in the IBM Data Analytics Accelerator (IDAA – formerly Netezza) that have been tied into the mainframe to speed time to resolve.

Callsign – a biometric authentication and fraud prevention technology, can leverage a modern smart phone to identify the owner/user of the device before they actually answer a challenge – which could be a finger print, facial recognition or voice. Using the accelerometer in the phone, the GPS and pressure points on the touch pad, along with historic behavior patterns, Callsign can tell by the way a person is holding a phone if it’s the original user or someone else before offering them the authentication challenge. This type of technology can be used at kiosks in regional/branch offices to enroll users and make sure they are the real person requesting later service. No need for a card. A unique user id is sufficient to provide authentication. True, many low-income users/beneficiaries may not have smart phone capability. Alternative mechanisms can be deployed for challenge/response authentication. But, maybe providing a low-cost device to beneficiaries for this purpose, a more modern version of the “RSA token devices”, might reduce overall costs for low-income users. Watch this space. One of the Callsign customers, a large credit card processing bank, is calling out to Callsign from a “legacy” mainframe transaction program to authenticate that the real customer is at the point of sale or ATM device requesting service. Compare that to an experience I had recently. Visiting 500 miles from home, I went to a big box department store and paid with a valid credit card. Everything was good, but the transaction was denied. I then used a debit card, same bank, same credit card service, but used my pin code. The transaction was approved. As I walked out of the store, I got a call from the credit card provider asking me if I just attempted to use the card. They restored my card to service immediately. Use of the Callsign capability eliminates the human intervention, lowers my embarrassment and speeds transaction processing.

Going a step further, Callsign runs on Amazon Web Services (AWS) or a private cloud today. This is a distributed connection to the transaction systems calling out to it. There are about 15 “risk tests” that can be done, but typically just three can be done and the results fed back to make a risk decision in the time allowed for a transaction to complete. We’ve hypothesized that if Callsign was running on a mainframe, with a memory connection to the transaction programs, that 10 risk tests could be done on the mainframe and maintain the service level agreement of the “legacy” transaction programs. Stay tuned for future updates in this pace.

The NSA has proven that leveraging a Google like search capability can help stop attacks. Why not use web crawling software to look for fraud and overpayments? Leveraging online obituary information, an insurance company or benefits providers could determine if a person has died and no longer eligible for services. In addition, it can predict the services that may be available to the survivors of that person. This can speed up time to deploy payments to their survivors. These web crawlers can feed a data warehouse searching for fraud but also feed real-time systems to avoid fraud for new transactions.

Collaboration is necessary to move forward:

Education: partnerships between vendors, businesses/agencies and schools is necessary to create the next generation of IT professionals (programmers and operations) as well as to update the skills of existing personnel.

Operations: Today, fiefdoms around individual architectures or administrative domains exist that create/foster conflict and drive up IT costs. Not everyone is going to get along. Organizational politics and budgets have as much to do with fiefdoms as anything. Leveraging the Rational developer example, where a small group of people have some hybrid responsibility, can lead to breakthroughs in processing schemes.

Legislation: Where necessary, this can be valuable to enable a leap toward something new that will provide value and reduce costs.

Summary

There is no right or perfect answer to any IT decision. As the saying goes and leading to an unintended consequence: “Throwing the baby out with the bathwater” isn’t necessarily a good approach. Leveraging a hybrid computing, operational and development environment can make a large shift toward leveraging “modern” application models. Happy programming!

Hybrid Computing

Emerging Technology and the role of the mainframe

November 14, 2014 Jim Porell 4 Comments

I recently was given a list of emerging technologies and asked how the mainframe, and in particular, z/OS, is relevant to those technologies. It’s a great question. Unfortunately, it’s important to understand the source and motivation of the question. Sometimes, the questioner is looking for an excuse to bury the mainframe. They’d have the unintended consequence of not finding or looking for synergy with the mainframe. In other cases, there is genuine curiosity. I’m going to go with curiosity in this case and give my best effort to respond.

Systems of Record are where data resides. Systems of Engagement are where information is accessible to end users. In many cases, PC systems are both a System of Record and a System of Engagement, so there is a thought that these “commodity” systems are “good enough” to handle all workloads. That couldn’t be further from the truth. Complexity, scale, security and business resilience are some of the problems that occur when commodity devices become the sole “solution” to problems. However, there is another major problem – operational silos. This occurs when one organization solves “one problem” while another organization solves and manages another business problem. Complexity and risk occurs when multiple organizations depend on each other to replicate data or share data for different purposes. This is where system security and resilience are at risk. It also requires duplication of data and duplication of effort to manage that data. Any duplication adds to costs.

I like a different approach. It’s based on leveraging the best of all technologies: commodity front end devices, such as PC’s, Smart Devices and the Internet of Things (definition later); Commodity servers for data transformation and hosting applications; large-scale servers for hosting and managing access to data, including a large amount of data manipulation and processing. In this hybrid environment, the goal is to bring the applications to the data. There will never be a single copy of data (e.g. backups, disaster recovery, cloning), nor will there ever be a single server to process that data. However, by sharing data and reducing copies, a simpler deployment model is possible. In addition, cross-platform security and resilience should be a part of the solution so that data is processed on a “need to know” basis and applications are highly available, end to end. It doesn’t make sense to have a back-end server (System of Record) that is 99.999% available if the front end infrastructure (System of Engagement) is full of availability and security issues. Our goal is to provide an end to end deployment infrastructure that provides efficient integration across the components and technologies necessary to meet a business’ workload needs. In the process, the business or government agency should dramatically reduce operational cost and complexity, while improving security and business resilience. This infrastructure can meet or exceed service level agreements and provide investment protection for the future.

Given the above context, here are a couple of emerging technologies and my view of where a hybrid approach can help them or not. I say my view, but the reality is, my good friend, George Thompson of IBM, provided the first pass at this list. We’ve been collaborating for many years. In my role as a consultant to Vicom Infinity, George is the principal IBMer I’ve been working with toward challenging customers to save $2 million dollars on IT expenses. So here we go:

1. Digital Security

There are many ways to look at this. But most importantly, collaboration is king. There will never be a “Single Sign On”. However, there are multiple sign ons with shared credentials. A good example of that is Apple has the same sign on credentials for it’s Apple Store, iMessage and iCloud, among other applications. But taking it a step further, they’ve introduced biometrics through finger print readers on their smart devices. Other forms of multi factor authentication can be deployed. There needs to be a source for “the truth”. There are several large banks and governments that have leveraged z/OS RACF for hosting digital certificates as the basis for authentication across applications and devices. One bank has stated that they avoid $16 million in annual license fees from third parties by hosting and managing their own digital certificate infrastructure on their existing mainframes.

Beyond authentication are digital footprints necessary for forensics utilized for Cyber security fraud, theft and rogue insider activities tracing. There are so many products that can collect logs and monitor those footprints. z/OS and Linux on z have been leveraged as collectors of these logs to allow for processing across workloads and to look for anomalies that might not be detected otherwise, if each different organizational unit was processing the request. The New York Police Department (NYPD) deployed a product from Intellinx that captured end-user activities across their agency. Up until that deployment, each of their 30+ applications had a unique home-grown audit capability. Intellinx enabled them to eliminate the “silo’ed” audits and combine them into a single commercial off the shelf (COTS) product. It also allowed them to find anomalies across the entire application suite that may not have been easily detected, manually, by their silo’ed offerings.

2. Virtual Personal Assistant

Cognitive computing is evolving rapidly. Speak into your phone or tablet. Ask a question or request a task be executed and your “wish” becomes their “command”. Simple requests are executed on the device itself (e.g. call a person). But many requests go to a central “cloud based” service. The request gets parsed for context, a knowledge base is queried and the result is provided to the requestor.

I don’t see the mainframe “operating” the Virtual Personal Assistant, at this time. However, I do see the mainframe as a source for the knowledge base for a wide variety of applications. If you ask to query your account balance, does the bank make a copy of that “up to date” business record and send it to the VPA server? No. The VPA is the System of Engagement. It translates the request into a query. The query is sent to the relevant server which processes the request and sends the results bank. The VPA then translates that into spoken word or some form of viewer by launching a device app to display the results. These are not mutually exclusive processes.
Going back to Digital Security, the back end server that processes the query could use the Digital Security provided by the previous authentication of that device. It could also send a challenge request directly to the device, as a form of multi factor authentication, to ensure it wasn’t a fraudulent request, such as a phishing attempt. Collaboration is critical.

3. Smart Workspace

I found an interesting definition at a Johnson Controls website. What was most interesting to me is that I know of Johnson Controls from their utility infrastructure monitoring devices, also known as SCADA or system control and data acquisition devices (e.g. thermostats, system monitors). And that leads me to think of the security of those devices. But I digress. Their vision is around Social Computing, Mobile and Collaboration so that the workplace of the future is a virtual office where you feel a sense of community with your peers rather than feel isolated. This includes document collaboration, video and screen sharing, smart walls/boards/screens that are touch sensitive. These can all be considered the System of Engagement. What’s the mainframe and z/OS role? The source of the files/documents. The authentication server that coordinates the integration of each of the pieces of this “virtual community”. The fault tolerant backup of the critical data elements. The workflow scheduler that “turns on” and coordinates the myriad of parts of a very large virtual community.

4. Software-Defined Anything (SDx)

The idea here is that instead of physical servers and physical devices or appliances, the IT world evolves to virtual appliances and virtual application images. The mainframe can be viewed for three major virtualization capabilities.

1. Logical partitioning – PR/SM LPAR – that enables the mainframe to be carved up into separate entities. This really is a physical partitioning, though and probably doesn’t apply.
2. z/OS – originally known as MVS for Multiple Virtual Storage – for it’s ability to run multiple applications and data types within the same operating system image.
3. z/VM – with its ability to run multiple operating systems and therefore, multiple application servers simultaneously and on demand.

Within z/OS and z/VM, there is software defined networks, memory and storage that enable direct sharing between workloads. In some cases, with new hardware definitions, only pointers to data are shared between applications to dramatically improve performance latency, reduce virtual and real memory and improve security, resilience and scale of the end to end workload.
That’s not to say that z/OS and z/VM are the answer to Software Defined Anything. The Anything is workload and solution dependent. These systems can participate effectively as part of a bigger solution to reduce costs and improve the solution qualities.

5. Affective Computing

This is an area that probably doesn’t have direct ties to a mainframe. As defined in wikipedia: it’s about computer science, psychology and cognitive computing. Think about robots that attempt to mimic human activity.

I don’t see arms and legs protruding from an IBM mainframe nor a mainframe chip within a robot, yet. I still see mainframe connections. One is through security. These robots need to be smart. They need to get their “smarts” from some source. That source needs to be secure. Authentication must occur. The robot will then “do things”. Are they transactional? These are things that can be done with a mainframe.

6. Neurobusiness

The Gartner Group definition is “capability of applying neuroscience insights to improve outcomes in customer and other business decision situations.” To me, insights equals analytics. Analytics on the mainframe and z/OS is fantastic. Why? It’s got the data. There are many businesses that look at trends, anomalies and other analytic insight to improve sales, identify fraud detection, forecast future trends and leverage existing data and analytic capabilities on the mainframes. There are also businesses that are running analytics on commodity servers against commodity hosted data and joining those results with analytics run on mainframes against mainframe data.

The definition, looking at other sources, is applied to training and decision making.I don’t necessarily see the mainframe participating in that aspect.

7. Prescriptive Analytics

Described by wikipedia as the third and final phase of business analytics (BA) which includes descriptive, predictive and prescriptive analytics. Vendors such as SAS and IBM’s SPSS have been on z/OS for many years. SAS has described the lack of value of “looking in the rear view mirror” rather than looking ahead to how you can get value for many years. The mainframe has plenty of the business data. These vendors have brought the applications to the data in order to gain the insight and provide the business value.

8. Data Science

A pseudonym for analytics. There are multiple parts of analytics:

The data.
The application that analyzes the data.
The application that presents the results.

The mainframe has long been excellent at hosting and processing the data. The System of Record. Data visualization is the role of the System of Engagement and commodity hosted devices. If a business wants to “copy the data” in batch, host it on a commodity server and then process it and display it on a commodity device, that’s their prerogative. But what does that cost them?

Time – necessary to make the copies.
Network – bandwidth to make the copies.
Storage – to host temporary and production copies of the data.
Compute Capacity/Scale – that’s used to move the data, instead of processing it.
Environmental – energy, floor space and cooling for the copies of data.
Money – for all this “excess” capacity.
And lest we forget – Security – to make sure that the “need to know” aspects of the particular data copied is handled the same, regardless of where it resides and
Resilience – back up capabilities for all the extra servers and data.

Needless to say, many businesses are leveraging near real-time analytics against the System of Record hosted on a mainframe and leveraging the best capabilities available on mobile devices and PC’s for the visualization of the results.

9. Smart Advisors

This is a System of Engagement. It could be a human or it could be the dissemination of data to a human. This is not the role of the mainframe. That Advisor needs to get its “smarts” someplace. As demonstrated earlier, the mainframe, z/OS, their data and their analytic processing can contribute to those “smarts”.

10. Speech-to-Speech Translation

This is a System of Engagement. Not necessarily the role of a mainframe. Once it’s translated to actions, the mainframe is happy to oblige and process the request. It can also work to ensure the authentication of the user/device requesting the translation, when necessary.

11. Internet of Things (IoT)

This typically refers to the myriad of new devices that are being made accessible via the Internet, e.g. Home appliances, Light bulbs, cars, cameras, etc. For this reason, the Internet naming convention was running out of addresses (IPv4), so a new addressing convention, IPv6 was created to handle the demand. Most of these devices are Systems of Engagement. They need to securely access Systems of Record.

The mainframe and z/OS have introduced IPv6 to enable direct connection to those devices. As defined earlier, the security of those devices and the monitoring of them can be handled on a mainframe. These devices are not islands, nor is the mainframe. They can easily collaborate to bring the best of all worlds into new solutions and other emerging technologies. IBM recently launched a foundation for the Internet of Things.

12. Natural-Language Question Answering

At the front end, this is a System of Engagement. To get the answers, Systems of Engagement and knowledge bases are required. I don’t envision the mainframe doing the natural language parsing, but I certainly envision their role in preparing the answer and securing the connection from end to end.

13. Complex Event Processing (CEP)

I love math. Patterns, Fractals, Recursion. Macro and Micro views. Computers are outstanding at repeating patterns. Many of the problems that I see in contemporary society are because we are too close to a problem. If you step back a little, a look at “the bigger picture”, you can see patterns repeated.
Event processing is like a Dispatcher. Every operating system has a dispatcher at a kernel level. Many operating systems cannot dispatch disparate work simultaneously because those systems don’t know how to balance the needs of the many and prioritize them against the needs of a few. Deadlocks, overcommitment and race conditions occur for unsuccessful systems. z/OS and the mainframe have demonstrated excellent capabilities for avoiding these issues and provide granularity, at a business level, for balancing the processing needs. More importantly, many years ago, they created workflow processing that deals with the success or failure of prior tasks to determine the next task. At a micro level, z/OS has been executing Complex Event Processing for decades.
Now, the term applies to end to end business solutions that include multiple Systems of Engagement and multiple Systems of Record. There are several middleware solutions that enable the mainframe and z/OS to participate and to manage CEP and be managed by CEP across a disparate group of systems and devices.

14. Big Data

This is primarily dealing with the System of Record and the analysis and processing of the data within Systems of Record. z/OS and the mainframe have demonstrated the ability to process their own data, the data of other systems and to have their data processed by other systems.
Security, storage management and resilience of the data on both z/OS and other servers can be managed from z/OS as well.

15. In-Memory Database Management Systems

As a database, this is a System of Record. For many years, middleware, such as CICS for z/OS, has provided an in-memory database management system. New applications are looking for SQL and other industry standard interfaces to these in memory databases. z/OS and the mainframe are capable of meeting these needs and working in collaboration with other systems that provide these capabilities. In addition, z/OS and the mainframe can provide authentication and resilience services for these databases.

16. Content Analytics

Content analytics is the act of applying business intelligence (BI) and business analytics (BA) practices to digital content. Companies use content analytics software to provide visibility into the amount of content that is being created, the nature of that content and how it is used.
This is another area of System of Record. z/OS and the mainframe have a variety of middleware associated with content management, including archive functions for media streams, documents, and other non-relational data types. In turn, there are analytic solutions available on the mainframe and off it to process that data.

17. Hybrid Cloud Computing

By definition, this includes internally managed clouds in concert with externally accessed clouds by a business or agency. zEnterprise is the premier Hybrid Cloud platform supporting Public, Private and Community clouds across heterogeneous architecture and supporting many cloud infrastructures including OpenStack. Enough said.

18. Machine-to-Machine (M2M) Communication Services

This is related to the communication between the devices associated with the Internet of Things and the success of IPv6. As stated earlier, the mainframe and z/OS are already enabled for this form of communication.

19. Cloud Computing

Somewhat redundant with the Hybrid cloud computing item above, both z/OS and the mainframe provide cloud hosting capabilities. Allen Systems Group’s Cloudfactory/Mainframe has enabled much of the functionality of z/OS to be accessed and provisioned via an interface that is similar to Amazon Web Services.

20. Gesture Control

By definition, this is a human computer interface and therefore a part of the System of Engagement. This is not a role that I envision the mainframe to undertake. The gestures will be interpreted and actions are taken, as a result. Some of these actions may be directed toward transactions or applications hosted on the mainframe and z/OS.

21. In-Memory Analytics

There are several approaches to solve this problem. Architecturally, there are some valuable differences.

For the x86 and Power architectures, IBM has delivered:
IBM’s DB2 10.5 with BLU Acceleration, typical queries in an analytics workload have been shown to be more than 1,000 times faster than other leading databases. Innovations in BLU Acceleration include:
·        ‘Dynamic in-memory” columnar processing providing not only dramatic analytics performance – up to 25 times faster -– but also the ability to scale for expanding Big Data needs without the limitations imposed by traditional in-memory systems.
·        “Load and go” simplicity which allows clients access to blazing-fast analytics transparently to their applications, without the need to develop a separate layer of data modeling.
·        “Parallel vector processing” for high-performance data analysis in parallel across different processors.
·        “Actionable compression,” providing as much as 10 times storage space savings where data no longer has to be decompressed to be analyzed.
This DB2 is typically called LUW – Linux, Unix and Windows version. In this case, the BLU acceleration is not available on Linux for System z implementation of DB2. However, through database connection middleware, the z/OS data can be accessed by this product.
Hadoop is an open source implementation of a large scale analytic server that has been deployed on the mainframe by IBM and by Veristorm’s zDoop offering. This can leverage most of the z/OS System of Record databases and make them accessible to the Hadoop File System running on the mainframe or other Hadoop servers.
Another hybrid implementation is the IBM Database Analytics Accelerator (IDAA). This is a co-processor that works with flash (memory) copies of data on z/OS and can process a query 1000x faster that z/OS might on it’s own. There are several operational benefits to this approach:

Security – the authentication, access control and logging are all done in the context of the z/OS user that initiated the query request. This simplifies audit and analysis of user behaviors.
Latency from OLTP to Analytics – It provides near real-time access provided to transactional data where other systems might be using a copy of time that takes a lengthy time to unload, transfer and reload prior to its availability for analytic processing.
Cost – it provides commodity analytic server costs without the need for extra management as an independent server instance and the costs of security and resilience associated with that.

22. Activity Streams

This are generated at a System of Engagement. Activity streams are generally associated with Social Media applications. There are a number of products that will take these feeds and coordinate them across Social Media platforms. I am not aware of any of them running natively on z/OS, though I do believe the IBM Connections can run on the mainframe. Some of the z/OS management activities can be processed as activity streams and posted to social media. This is valuable where an internal wiki might be used to manage or display mainframe system status.

23. Speech Recognition

Speech recognition research and development has been going strong at IBM for almost 50 years. Throughout that time, more than 200 IBMers have contributed to the significant advancements in this field. That being said, it is a System of Engagement.
Middleware is available, as part of multi factor authentication, to leverage speech recognition and patterns as a login method that can be passed to the mainframe. Speech recognition middleware can also be leveraged to begin applications or start tasks on z/OS and the mainframe. Customers have used these techniques to simplify the human interface to z/OS.

That’s the end of the list I received, but not the end of my thoughts on Emerging Technologies.

The Evolution of the Mainframe.

Database Processing.

By nature, database processing deals with the System of Record. IBM’s DB2 platform was originally deployed in the early 1980’s after a successful research product. Later, on a separate code base, the DB2 for Linux Unix and Windows was created with a similar programming interface to the mainframe version. Mainframe customers demand consistency and a legacy that they can count on. Once implemented by IBM or other vendors and then successfully used in production by a customer, the customer expects that code to run “forever”. I can tell you that I’m working with a customer now whose original code was written in 1969 and they are many generations of hardware and software old and out of service, but they are still working with integrators to keep it running.
With those requirements in mind, IBM has developed a philosophy that many of the new technologies will be deployed on the DB2 LUW version first. If successful, that functionality will be later integrated into the DB2 for z/OS version. If it is unsuccessful, there is no harm in not adding it to the mainframe version as it might be just a niche offering.

Evolution of new technology

Many of the System of Record emerging technologies listed in this blog will share a similar fate. Where they have not been implemented on the mainframe yet, they will be considered for the future based on their customer exploitation merits. You’ll notice I used a phrase: “not on the mainframe, yet”, a couple of times above. That’s because I believe it. Some of these emerging technologies will become ubiquitous and demand their place on the mainframe in much the same way as the DB2 technologies have evolved. Think about TCP/IP, Linux, Java, and XML that were once emerging technologies and are ubiquitously deployed on the Mainframe and within z/OS. Even Linux inside z/OS…at least some of it.

Same code. Different Container. Different Operations Model

With the adoption of open source and open programming interfaces, there are few programs that can’t be deployed on z/OS or the mainframe. But just because it can be done, doesn’t mean it should be done. For example, z/OS is branded as a Unix system because of the Unix System Services component. By that brand, it means it can support a VT100 character based terminal as an input device. So, using the vi editor, if you type a character, it would immediately get sent to the mainframe for processing. Type the next character and the same response. Move the mouse and you burn mainframe mips chasing it. Yes, that works, but it is a complete waste of mainframe processing. Capture the edit on a PC or smart device, using local processing. When done or the user hits enter, send the bulk of the input to the server to be saved and processed. Data processing is what the mainframe is about. The punch card and the 3270 terminal are “old school” systems of engagement. The mainframe has adapted to those interfaces as well. The z/OS Management Facility is a new web services based implementation that can augment or replace the “old school” 3270 command line functionality. New products, such as IBM zSecure and IBM Wave put a graphic front end on the mainframe. It is collaborative and hybrid deployments such as these vs. replacement of a legacy.

What’s your cluster look like?

The mainframe Parallel Sysplex solved clustered computing in a dramatically different fashion than commodity servers. Rather than separate data into smaller consumable chunks that are then spread across database servers that are then attached to clusters of application servers that need to have round robin workload balancing for some semblance of security, the mainframe did it different. They decided to share all data across their “cluster”. Each system has direct access to the data much like a SAN. This access is now Fiber Channel based, via the FICON protocol. It runs on the same fiber optics wiring as the FCP protocol, but it has better latency, security, error correction and redundancy. Shared by these clustered servers is the Coupling Facility. This is a separate mainframe server or logical partition with three functions:

High speed communications (peer-to-peer) between the “cluster members”.
Lock manager for read/write access to the shared data across the cluster.
Data cache for the most recently used data (in memory) to avoid disk access for high volume transaction processing.

Commodity database server developers are beginning to make their own “coupling facilities” with a fraction of the functionality, performance, scale and reliability built into the mainframe Parallel Sysplex capabilities.

System Integrity

In 1973, IBM issued an integrity guarantee that any problems found in their code on the mainframe that could, and I’m paraphrasing, give an unauthorized person an undeserved authority, promote a program from user space to kernel space without authority or could manipulate or destroy data without authority would provide a fix at no charge, as quickly as possible. Recently, the guarantee was updated. But there is a reason for the guarantee and that’s baked into the hardware and software architecture of the mainframe. The hardware creates a boundary and set of instructions to manage the transition between user and kernel (system) processing. It also enforces boundaries on memory within the operating systems. As a result, a “buffer overflow” between user space and kernel space will be detected by the hardware. So in case there was poor programming on behalf of the operating system or middleware, the hardware is smart enough to detect the error and abnormally terminate the offending user program. It’s not to say the mainframe is hacker proof. Better stated, it is hacker resistant. The hardware architecture will detect and inhibit the majority of problems found from poor programming on commodity systems. In fact, execution of “portable” code on the mainframe has found a number of integrity problems that might have gone unnoticed on commodity software.

Multi-tenancy

Baked into the mainframe hardware and software is the goal that multiple applications, databases, and for that matter, operating systems, can run simultaneously without negatively impacting each other. The impact being integrity and performance based. The mainframe workload management capabilities for managing service levels and scale are legendary. Many more eggs can be put in a single basket. Far few servers need to be deployed. As PC servers became rampant in data centers, VMWARE came along and began to offer up to 80:1 reduction in the number of physical servers required to deploy workloads. Simultaneously, z/VM and Linux might offer an 800:1 reduction of the same workloads. Operational fiefdoms being what they are, an organization might be satisfied with the savings of an 80:1 reduction. Working collaboratively, there might be a 400:1 reduction in servers, with some on virtual blades and some of the mainframe.

Summary

This is not an either-or proposition, though organizationally, it might feel that way. Collaboration and sharing is at the foundation of the mainframe architecture. Now, through networks and multiple servers, that collaboration extends to Systems of Engagement.

A truly modern IT organization can realize the benefits of collaboration in application development, business resilience, time to deployment, operational risk and simply put, cost savings.

All of the above examples are to prove the point of the value of hybrid and collaborative computing. There are many offerings that provide similar value to those listed in this post.

Hybrid Computing, Mobile Computing

Secure Hybrid Computing – White paper

October 1, 2014 Jim Porell 1 Comment

I’ve written several times about Hybrid Computing, data patterns, reducing costs, etc. Well, it’s all wrapped up in a White paper now, available at this site. HybridComputing.CaseStudy.Whitepaper.V1

Readers of this blog know that it is mainframe centric. They might be surprised that there is no mainframe nor, for that matter, any other platform mentioned in the paper. Why? It’s not about technology. It’s about business. Saving money and growing profits. If that doesn’t get attention, I don’t know what will. When systems are built together collaboratively, instead of fiefdoms, any business can save on a wide variety of server platforms.

In the paper, you’ll see that IBM and Vicom Infinity are working together with infrastructure that can be used to demo or develop proof of concepts to put these ideas to work. Don’t hesitate to reach out to me for additional information.

So get started thinking differently. You can start with this blog and white paper. Happy programming. HybridComputing.CaseStudy.Whitepaper.V1

Hybrid Computing, Unix System Services

If you started with computers in the 1990’s and hated mainframes – you were right

September 2, 2014 Jim Porell 4 Comments

Wow, I said it. It’s hard to admit for some of my peers, but looking back, there is a lot of truth to that sentiment. I’m not talking about traditional transaction processing used for insurance claims, point of sale, ATM’s, travel reservations, etc. That’s where money is being made and continues to be made, with many businesses leveraging mainframes as their System of Record.

No, that’s meant for folks that were and are interested in client/server deployments. Folks that wanted to use modern application development tools. Folks that wanted to incorporate multi-media and streaming into their workflows. These might be considered the System of Engagement today. Oh my goodness, the mainframe was not at all prepared to handle that work. Decisions were also made for departmental computers. Can you imagine a mainframe of that era in a closet? A closet? Where was the chilled water, the raised floor, the humongous air movers?

It was a traumatic change within IBM to begin the modernization of the platform and 1. keep it relevant to where it was having its “legacy” value (the prime objective) and 1A. make it relevant to capture new workloads. This was something tolerated, but not completely embraced at the IBM executive levels. Ignoring the success of “desktop computing” and new business opportunities, I always felt that the mainframe sales mantra was kind of the opposite of the famous Star Trek line: To boldly go where we have already sold before. So the real goal appeared to be to keep our current mainframe customers happy.

The mainframe is a dinosaur. The mainframe is dead

There were voices, everywhere, saying the mainframe was dead, it’s a dinosaur. It chilled the hearts and minds of senior IBM executives too. There was a “new breed” of executive that wanted to look and grow new business lines, like the PC Server and Power Systems. Where did their funding come from? The profit of the mainframe. But as a result, mainframe R&D budgets were challenged. And even within the mainframe, as growth opportunities were considered, the development budget was spread even thinner across a wider variety of efforts. Some believed it was a cash cow, from which new opportunities should be funded. And then there were the mainframe believers that had to fight “the new status quo” to maintain their budgets. I could go on and on about these political battles inside IBM. There’s some TMZ quality stuff, but not what I want to discuss here.

Now, if you don’t want to read the story about what changes were made and how they were made, you can skip to the Summary of how bad it was and how good it has become.

The long road toward making the mainframe relevant to new workloads

Instead, I’d like to tell you about the changes that were made to make the platform relevant to new business opportunities. In the process, the positioning of the mainframe changed. It’s still a terrific System of Record, but now, unlike the 1990’s, it can be a viable System of Engagement for many functions, but not all. Because if I said all functions, I’d be lying every time I mentioned hybrid computing!

Let’s start with a big mainframe app that made the decision to go “all distributed” and never looked back. That hadas much to do with money as it did with technology.

Dassault never looks back. Goodbye mainframe

Dassault System’s CATIA application – a CAD/CAM – engineering modeling/rendering application. They were charging $8K a user for the MVS version. Each user had a specially built 3270 Graphics Adaptor for model renderings on a “green screen”.

There were so many instructions required to do the renderings that each terminal had it’s own hardware interface. At the time, there were only 4096 interfaces (called Unit Control Blocks – UCB) on the system. As more engineers created modern airplanes, the customers had to buy more systems, which were not cheap to acquire, nor operate. Ultimately, the z/OS operating system eliminated the constraint on the number of UCBs, which would have greatly helped CATIA customers, but had the added benefit of helping all customers when Parallel Sysplex was introduced later.

CATIA grew bored with the proprietary 3270 GA box as their graphics rendering device. They wanted color and simpler graphics.

This is just what the UNIX community was demonstrating. So they created a new product, based on AIX. They thought the UNIX based version should charge about $10K a seat. Some cooler graphics were worth the price, but not too far off the current customer price. They did a market study. They found other UNIX based engineering programs were getting about $25K a seat And those systems were selling well. What a profit margin! So they decided to charge $22K a seat, in order to undercut their competitors. They also yelled from the rooftops “the mainframe is dead” in order to transition to the new model, but more importantly, the new revenue and profit stream. What was surprising to me was that Dassault was partially owned by IBM. My job was to talk to the “chief yeller” and convince them of the “new mainframe strategy” which was still vapor ware, from their perspective. All requests to stay on the mainframe fell on deaf ears, and rightfully so, in hindsight.

SAP tells IBM what’s wrong with the mainframe

A “big mainframe” app that transitioned to commodity systems was from SAP, which was mainly a bunch of ex-IBMers that had a really good idea. SAP R/2 was created with a mainframe back end application and database server which also drove the green screen terminal front ends.
SAP R/3, the next major version, was a “client/server” app that was all about the end user experience (GUI) and commodity server deployment models. It didn’t run on the mainframe when it first came out. SAP was one of the voices bragging about the “death of the mainframe” in order to transition customers to their new product. It took years of negotiations and modernization of the mainframe infrastructure and pricing changes for SAP R/3 to “return” to the mainframe, but it was only a partial return….The Database, entirely, and then the data access methods and some of the applications. The presentation layer has expanded beyond the desktop into mobile and web services. The original application programming language remains the same, though it has been augmented with Java.

Let’s peel the onion and look into modernization specifically for SAP R/3. In my opinion, it was that application, or better stated, that company that forced IBM to change the mainframe. IBM wanted new workloads. Here was a big one that took business away…Let’s win it back.

Network Performance and System Integrity

First, client/server was based on the internet and local area networks. They were rapidly transitioning to TCP/IP hosted networks – an open network. IBM was still stuck on SNA, it’s proprietary network that was actually a de facto industry standard as most systems could interoperate with it. Today, SNA applications can run over TCP/IP….it’s about layering, but that’s too detailed. An implementation of TCP/IP for MVS is created. Check the box. Done. Well, wait a minute…how well does it perform, SAP asks. It takes about 186,000 instructions to send a small chunk of data out and receive the acknowledgement of successful delivery. By contrast, it takes AIX about 18,000 instructions to do the same. So MVS is about 10 times slower communicating with TCP/IP than other platforms. Well, no it’s not. The SAP R/3 application is very chatty. The application server makes a network connection to the database server on the mainframe. For each end user request that ends up making a database call, there are actually 26 send/receive pairs for each transaction. So the overall pathlength is 260 times worse than AIX for a single SAP transaction. At the mainframe hardware and software price levels, that isn’t going to sell well. I also made a joke of this at the time. What’s Lassie, RinTinTin, Benji and TCP/IP on MVS?

Three movie stars and a dog. Drum roll please. This was a bad situation. Unfortunately, this joke lasted several years. A lot of effort went into looking at alternative implementations.

Anytime there is a network, data must move from one system to another system. In a terminal environment, such as a 3270 screen update, it might only be 100 bytes of data get transmitted. But a video could be millions of bytes. IBM, from it’s outset, has been historically great at creating architectures. Some architectures have layers of functionality. Communications or networking architectures are famous for the seven layer diagram,

where each layer has an architectural purpose. Application layer, transport layer, session and network layers being a subset. IBM has an image processing application originally developed for medical records called Imageplus. The performance of image processing (and later streaming) over the network, because of the rigidity and adherence to the layers of architecture by the original code base, was horrendous. The physical image was COPIED TWELVE times, once for each layer and a few times within the application, as it traveled through the operating system. Eventually, the networking software was modified so that it could copy the data from the application buffers directly to the transmission wires after a quick test of the applications data integrity (aka make sure there would be no buffer overflows or attempt by an application to send system or privileged data that they didn’t have authority or “need to know”. This data integrity testing, by the way, is inherent throughout the mainframe operating systems and middleware).

So the Imageplus benchmark was also critical to understanding how to re-write the TCP/IP stack. If the system knew the SAP Application server was running on an AIX server that was channel attached to the mainframe, the new performance was 5,000 instructions. (Didn’t need to do some network error handling nor routing and this saved instructions). If the Application server was attached via a router, then it was 10,000 instructions. Hallelujah….we were better than the rest of the world…..

Data Character Translation

Not yet, says SAP. Those desktops, those application servers on Windows and UNIX/AIX? They like to read and present data in ASCII format. Your mainframe and DB2? They like to process data in EBCDIC format. It takes seven instructions to translate each byte. So all that network gain that was just achieved is lost by data translation. As a result, effort is put into the path length of data translation. Eventually, it gets down to three instructions per character. With the network savings, this is roughly equivalent, end to end of a commodity system performance. With price/performance, it’s still not good enough. As a result, the mainframe hardware, operating system and DB2 middleware are changed to natively support unicode characters, wide characters (for character based languages) and many other code pages. There was, and is, no longer a penalty for code conversion. Finally, after several years of work, SAP R/3 sales of hybrid computing began.

New workload pricing

But the transition isn’t complete. The introduction of Linux on System z leads one to believe that the mainframe is a viable application server. Not so fast, says SAP. IBM charges software by the MIPS or capacity of the ENTIRE machine. So if a customer adds one processor to run Linux on a mainframe that has 9 existing processors, the Linux license charge would be for 10 processors worth of work vs. the single processor that is executing. I’m going to rapidly jump ahead, but specialty engines were created such that new workloads, such as Linux, JAVA on z/OS, distributed connections to z/OS databases, some z/OS system utilities and more are charged by the actual engine or capacity of the workload vs. the entire capacity of the machine. Finally, incremental workloads could be added to the platform at or near commodity prices for the equivalent work. But we’re still not done with this evolution in hybrid computing.

Stop copying data. Share memory instead

Avoiding data moves was critical to the success of the “new” TCP/IP within z/OS. So next, the same concept is applied WITHIN the mainframe. And this takes several iterations with continuing benefit.

In a Blade server or rack mounted system, a special back plane or Top of Rack switch may be deployed so that communications within a ‘server box’ can go point to point, ala the mainframe channel to channel interface and avoid router overhead. Under z/VM with hundreds of Linux images running and then z/VM to z/OS within the same physical server, IBM created the hipersocket which uses dedicated hardware memory, instead of wires, to facilitate intra-physical server communications. The most recent announcement upgraded the hipersocket technology to leverage RDMA memory that can be shared between server images so that only a pointer to the data is transmitted. Each system can have direct access and eliminate intra-server copying of the data.

Increased capacity and memory available for new workloads

All of the above mentioned activities result in fewer data moves within the operating systems, virtualization layers and hardware servers themselves. This avoids instructions, real memory and latency required to do data moves. This allows the processors and memory to be available for other workloads. This allows greater scale of the environment. And because the mainframe has been doing this for over 40 years as a balanced architecture across instructions, memory, networks and data, instruction pipe lining, many levels of memory caching and more, it is capable of putting, colloquially, 10 pounds of “stuff” in a 5 pound bag without fear of breaking. Said another way, there is no fear in running the system at 100% utilization for very long periods of time.

The “Legacy” workloads get modern as well

So that’s just a snippet of the technology that went into “modernizing” the mainframe for new workloads. There is significant other infrastructure, such as the Parallel Sysplex, Geographically Dispersed Parallel Sysplex (GDPS), evolution from BiPolar to CMOS technology, reductions in cooling, electricity and floor space, packaging of “spare” parts in the box for fail over/fault avoidance and on demand deployment for capacity upgrades and more that made the mainframe better for traditional workloads, and over time, those benefits have applied to the new workloads as well.

But then, how much does it cost?

Historically, the mainframe has been considered an expensive alternative to “commodity” platforms that are “just good enough”. Much has been done to change the pricing structures.

Technology Dividends

Since the introduction of OS/390, IBM and other vendors have been providing tech dividends such that the price/mips has decreased regularly and with most new processor introductions.

Capacity Backup

The “z” in System z, stands for Zero down time. While not reality, it is a goal, both at the technical level and through software license charges. IBM introduced Capacity BackUP (CBU) pricing for backup or disaster recovery servers. A fraction of the production price is paid for the hardware server and no software license charges are paid. Many other vendors have accepted this model as well. Why? Because a business isn’t getting any productive work out of the backup servers. When production work moves over to the CBU server, then the software license charges transfer to that machine. A business may do several “fail overs” a year, to test recovery operations without incurring a license charge. When compared to “commodity” servers, this can be a tremendous savings. In those environments, multiple servers may be required and each of those will be charged a production license fee.

Development pricing

If a business wants to get more workloads, then it needs to cater to application developers. Those developers need a sandbox or low priced system to create that code. Unfortunately, software license charges being as they are, all MIPS on the mainframe were created equal and treated as production. Rational was a brand acquired by IBM. They had created some fantastic tools that worked across platforms and now included OS/390 and later z/OS. Those applications would require CICS, IMS, DB2 and other vendor middleware as the target production environment. However, IBM was charging the same price for the production licenses. Most other vendors, including Microsoft, Sun, Oracle and HP were giving away the run times when they sold their development tools. As a result, the mainframe was not targeted by most vendors as a viable deployment platform because the “cost of entry” was way too high.

IBM finally released a mainframe architecture that ran on PC and Power servers – the z Personal Developer Tool (zPDT) to vendors. This created a competitive cost for vendor developers to target the mainframe as a production platform. It was several years later that IBM decided to make this available to businesses or customers. Finally, there was a common tool set, Rational Developer and free run times and test environments that were priced competitively with “desktop” development tools. Better yet, a developer could “take a mainframe home” with them, as the zPDT could run on a Thinkpad laptop.

Parallel Sysplex pricing

Clustering computers is an important way to scale. IBM invented the Parallel Sysplex architecture that accomplished three things, with two being a huge differentiator from “commodity” server clustering.

Load balancing work across the cluster
Shared Lock management of database and file records across the cluster
Shared cache of the database and file records across the cluster, allowing all cluster members to have direct access to the data.

The direct access of shared locks and data cache allow additional servers to be added to the cluster without having to reorganization or partition the data. The performance is such that there is linear scalability as each new cluster member is added to a maximum of 32 systems.

Software was then discounted for customers that chose to use the Parallel Sysplex as a clustered model over a multi system model that had non shared data.

New workload pricing

This was discussed earlier. The net is that specialty engines were defined that allowed no software license charges or deeply discounted license charges for Java, Linux and other workloads.

Hybrid Computing pricing

With the zEnterprise servers, the mainframe added the ability to have direct attachment to an IBM Bladecenter and in the process, have a dedicated communications channel and a dedicated management channel for these two devices. I already described how performance can improve when channel attached when discussing the SAP R/3 deployment between AIX and z/OS.

The IBM Data Application Accelerator (IDAA) is a query server, running on x86 blades that can get direct access to DB2 on z/OS. By shipping queries over to the IDAA server, read/write operations can continue on z/OS while read only performance of the query can be parallelized across multiple x86 engines and see performance improvements that could be 300 to 1000 times better than if run on z/OS. More importantly, the IDAA is “just an engine”. Security of the database and audit remains within DB2 for z/OS. Data copies over the network can be avoided. Some copying is inevitable, but flash copies can be made in moments, instead of long running data extract, transform and loads (ETL) to other platforms. This provides a significant advantage to completing near real time analytics and enables new decisions to be made, prior to completion of a transaction. There are too many cost savings to mention, but some are: less disk space, faster response time, improved security, better scale, less network bandwidth consumed, etc.

Mobile pricing

In the 1960’s and continuing through the early 1980’s, it was agents of a business that executed transactions. Travel agents, Bank Tellers, Claims adminstrators, ATMs and Point of Sale terminals. The consumer watched. These transactions were typically short, easily measured and predictable. The majority of the transactions occurred during business hours, such as 9AM to 5PM. Hardware and software were priced for the size of the machine dictated the overall software license charges.

The “Internet of things” has changed that. Consumers can now do things on their own, where an agent was previously required. They can start a claim, transfer money, deposit a check, buy tickets to performances and for travel. They can do this at any time of the day or night. They can query prices as often as they wish, waiting for the “sale price” to be good enough to actually purchase something. This is dramatically grown the number of transactions being executed and kept the Systems of Engagement and Systems of Record up around the clock. Any down time means potential and real loss of business. In April of 2014, IBM introduced the Mobile price usage to provide a discount for these type of consumer issued transactions. The goal is to make the monthly pricing more predictable and comparable with “commodity” platforms.

Summary

How bad it was (1990’s)

Let’s review how bad the mainframe was back in the early 1990’s.

Water cooled, very large, heavy servers with large amounts of electricity and cooling required.
An Internet network that was horrendously slow at the hardware and software levels.
A Green screen Command oriented interface (similar to the DOS Shell that lived a very short while on PCs). Yes, you could “screen scrape” to make it look more graphic, but many considered that “lipstick on a pig”.
A communication architecture that was inherently inefficient as it copied data between system components, many times, before it went on “the wire”. In PC LAN terms, it was a Ring 4 implementation (and worse) instead of Ring 0.
The wrong character set was used: EBCDIC, which required software changes and data conversions to and from ASCII or Unicode.
Expensive software licenses for production and development.
Proprietary, green screen oriented, development tools
Proprietary programming interfaces
Lack of Commercial Off the Shelf (COTS) new workloads

How much better it is today

Comparably sized servers to commodity systems that use less electricity and cooling.
An incredibly faster Internet connection for inter-system, intra-system and intra-cluster communications.
While the command line interface is still available, a web service oriented management interface is now available.
A communication interface that passes pointers to data and shares the data rather than copying it within a system image, across virtual system images and across a cluster of systems.
Adoption of Unicode, ASCII and EBCDIC as base characters simplifies data consolidation on z.
Technology dividends and a wide variety of pricing options make the TCO and TCA of System z competitive with commodity servers
Single, cross platform tooling from Rational that includes mainframes, UNIX/Linux, Windows and Mobile systems (via the Worklight acquisition)
Portability of applications within z/OS and Linux in a variety of open languages
A far broader range of COTS software is available for Linux on z and z/OS

Some additional items that are better than commodity servers

Shared data access, with integrity, scale and resilience, across systems
Shared Analytics and Transaction Processing to a single database (that can be shared across a cluster), while maintaining Service Level Agreements
Hacker resistant (not Hacker proof) architecture that inhibits data and buffer overlays. The System Integrity guarantee has been in place since 1973.
Capacity Backup licensing and acquisition dramatically reduce Disaster Recovery costs and procedures.
System z hardware avoids 80% of the errors that might occur in a PC server environment. No down time nor failover would occur. System z software and hardware work together to drive System availability to 99.999%.
Workload management of thousands of applications and hundreds of thousands of client connections enables dramatic cost savings over alternative servers.
Incremental additions of software workloads without the need to install new hardware due to on board “spares” available to be turned on, on demand.

Summary

If you haven’t considered a mainframe in the last 20 years, it’s quite understandable. But if you don’t start reconsidering it today, you are making a fundamental mistake.

The modern mainframe is greatly simplified from where I began 40 years ago. I’m happy to say I may have had a little bit to do with that 😉 Happy programming.

Hybrid Computing

Modern Data Usage Patterns – a case for Data consolidation

August 28, 2014 Jim Porell 3 Comments

Last week, I wrote about organizational fiefdom’s and how they can inhibit efficiency in deployment models.

This week, I’ll describe a couple of data patterns that are common across several business models. Most important, they can take advantage of a hybrid deployment model and some unique System z characteristics that can result in a dramatic reduction in operational and security overhead and simplify compliance to a wide variety of government, industry and business regulations. It’s all based on a shared data model and collaboration across end to end technologies.

To me, there are three critical business oriented data operations: update a record, read a record and analyze a collection of records. There are also management operations: backup/archive, migration/recall and disaster recovery. I’m going to focus on the business oriented aspects for this post.

Let’s consider three different scenarios. A national intelligence operation that is processing satellite and other electronic information. A health care environment that processes medical records and data from medical devices. And a Criminal database containing wants, warrants and criminal records.

Each of these has a data ingest process that comes from individuals or individual devices. Satellite data is beamed to earth, typically to an x86 based server and then transmitted and loaded into a “System of Record” which might be considered the master database.

Medical records can be updated by a medical professional and patient via an end user device or portal and input can be received from medical devices e.g. EKG, MRI, XRay, etc. All of this information is then loaded into a master database.

BOLO’s (Be On the Look Out), Criminal Records, Wants and Warrants are input by various police agencies and transmitted to a master record database that can be accessed by other police departments to see if someone they’ve stopped or is in custody may be wanted by other jurisdictions.

Each of these scenarios has something special about them – a need to know. A doctor or nurse can’t “troll” a medical database looking for any data. That’s against HIPAA policy. They should only be looking at records associated with patients they are working with.

Intelligence analysts may only be able to see certain satellite or ELINT (electronic intelligence feeds) based on their security clearance.

Police in one jurisdiction cannot query or update records in other jurisdictions unless they are pre-approved for a particular case.

This Need to Know can also be called Compartmentalization or labeling of data. DB2 on z/OS has technology known as Multi-Level Security that allows data to be hidden from users and applications that don’t have a need to know. The best part of this technology is there doesn’t need to be any change to an application. The need to know criteria is established between security administrators and database administrators. As a result, when multiple users attempt to query an entire database, if they are in different compartments they’ll get completely different result sets without knowing the full breadth of the database.

So let’s look at a Medical System that has multiple hospitals scattered across a broad geography. Each hospital has specialty areas: Orthopedics, Oncologists, Pediatrics, etc. There is a Primary Care Physician (PCP) for each individual patient. There is the Patient. There are a bunch of different medical test devices: MRI, CTScan, EKG, etc. At some point, a Patient Care Record is created. A PCP is identified to that Patient. They may order tests on behalf of the patient. Test data is captured, stored and linked to the patient’s record. A Cardiologist may see and annotate information associated with the EKG. Any other Cardiologist at the hospital may also see that EKG and annotate it. An Orthopedic surgeon may look at it, but not annotate it. A doctor at another hospital may not even know that the patient exists unless they are invited to look at it by a peer or via the patient requesting a second opinion.

The following diagram shows a Manufacturing business that gets a variety of “parts” from different suppliers. They allowed each of their suppliers to check the on hand inventory to allow for continuous manufacturing and improve the supply chain operations. The unintended consequence of this implementation was that each supplier could see another suppliers’ inventory and price per unit. As a result, devious suppliers could undercut the competition or worse, collude with their competitors to raise prices.

By turning on the labeled security capabilities of DB2, the suppliers can only see their records. Employees of the manufacturing company can see all the records in the database. No applications were changed. The manufacturing company had to collect some additional security information for each supplier in order for this to work properly. You’ll notice the inclusion of internet address (IP @) as a security context. The manufacturer can “force” supplier updates to come from the supplier’s site. It will not allow a supplier’s employees to logon from home, for example. This could help inhibit a rogue employee of the supplier from compromising the Manufacturer’s database.

There are other examples of production systems leveraging MLS capabilities. Lockheed Martin has been operating a secure environment for multiple agencies for many years. This has been for the National Geospatial Intelligence Agency (NGA) and its mission partners.

But here’s another important distinction from other models. The data operations are somewhat like the Eagle’s song Hotel California:

“We are programmed to receive, You can check out anytime you like… but you can never leave”.

That means if you are viewing the data, you are viewing the “System of Record”. Where you are viewing from is called the “System of Engagement“. By definition, the System of Engagement can overlay and complement the System of Record by transforming it. This is can be a stateless entity or read only. The XRay image, stored in a database, is just a collection of bytes. The end user may not have the proper viewer installed on their desktop. The System of Engagement will transform the image into something consumable and recognizable by the end user. The Hospital doesn’t make a copy of the data and transmit it to other service providers. If they did, they’d have to ensure that those new data owners of the copy adhered to the same stringent privacy laws for which they are accountable. This becomes a logistics nightmare. Instead, the “user” being a patient or medical professional accesses a program (The System of Engagement), which could be a virtual desktop or web service, which in turn accesses the database and remotely presents the requested data to the end user. This is more of an image as the end user device is considered stateless. No local copy is saved. Because this is solely a remote presentation of the information, that device can be exempt from Privacy audits because it is understood that no local copy is made. This doesn’t address an end user taking a picture or writing down information associated with the record being processed. There are other products that can be deployed to capture these breaches of privacy policy. I can cover that in a later post.

Compartments can be created that contain only a subset of the stored records, similar to a view. So analytic processing might be done across all database records, looking for patterns, fraud, opportunities, etc, but without including protected personally identifiable information. For example, disease outbreaks by region, trends, risks, etc, but again, this analysis may only be done by someone with a need to know.

Look here for a video associated with Intelligence Analysts dealing with Satellite transmissions and leveraging this workflow.

It may be hard for some of you, but imagine the different satellites are actually medical devices. Imagine the different compartments are associated with various hierarchies of users at a Corporate level, Branch/Region, Department and individual basis. Imagine the spatial data results of this video as demonstrated using Google Earth, are instead rendering views of XRays, CTScan’s, etc. Hopefully, this is a compelling view of the realm of possibilities. One thing that might appear contrary to what I described earlier is the fact that in this video, various users know that a satellite exists when they didn’t have a need to know. Some satellites may be Top Secret, so unauthorized users have no need to know that that particular satellite even exists. To correct that situation vs what the video depicts, if the System of Engagement had requested sign on by the user first, they would not have seen the entire list of satellite’s as that may have been excluded by an additional database query of accessible satellites for that user. However, when comparing to medical devices, there is no secret that there are multiple imaging devices, but the results may not be visible to a user, based on the need to know. Many, many options exist. These are just examples to get the discussion started about new possibilities.

There’s another important aspect of this. Any business with DB2 on z/OS already has the System of Record capability. There is a change in operations management required, but no additional software license charges required to implement this. Other platforms are required to separate data (aka copy it) to facilitate the ease of compartmentalization available on DB2 for z/OS. Analytics can be provided against this system of record locally, by products such as the IBM Data Analytics Accelerator (IDAA) or Veristorm’s zDoop, which is a Hadoop solution running on the mainframe. The mainframe is capable of meeting the service level agreements of both the updates and queries of the database with very large scale.

The Systems of Engagement may be Linux or Windows systems running on Virtual Desktops or PC servers, as well as within Linux for System z or z/OS application and transaction processing environments. The end user access could be from kiosks (thin client terminals), Smart Devices, PC’s or business specific devices e.g. Point of Sale, ATM, police cruiser access points, etc. These systems could be hosted in a public or private cloud. They could be part of an existing system infrastructure. Authentication and access control should be centrally managed across the entire operational infrastructure.

The net of all this is a couple of examples of hybrid computing and collaboration across systems that can dramatically reduce the complexity and improve the efficiency of end to end business processes. If you are still compartmentalizing operations by server silos you may have the unintended consequence of missing some dramatic cost savings or better stated, cost avoidance. Compartmentalization on a need to know basis may initially lead a business toward separation of duties and separation/copying of data. But with the capabilities described, it’s actually a form of consolidation and collaboration that enables a greater degree of sharing the System of Record. You might not have to spend more in systems deployment to solve some very complex problems. Happy programming!

Introduction

Server Fiefdoms Inhibit Optimized Business Solutions

August 23, 2014 Jim Porell 7 Comments

In the 1960’s, the IBM mainframe led a transition in business processing from a paper centric transaction processing environment to an IT centric processing environment. The combination of the Personal Computer and introduction of modems and later the Internet changed the IT community from being internally facing to customer centric computing. The introduction of the PC Server created “commodity centric” computing and, typically, folks running in that environment were against IT centric operations as the PC server could bring Department centric computing to individual business units.

The unintended consequence of all this was fiefdom’s were created to manage server silos. Over a decade of server deployments, individual IT organizations may have been created with business related names (e.g. Point of Sale org, Analytic org, Web hosting org, Claims administration org). The reality is each of these organizations might be dependent on a specific server infrastructure. As a result, the introduction of any other server infrastructure, for example mainframe to PC server or centralizing on a mainframe from UNIX or PC servers would be viewed as bad. The reality is no single server is capable of meeting all the IT needs of a business unless they are very, very small. And even then, multiple applications, which typically means, multiple server instances or operating system instances, will be required.

I am mainframe centric

I have no hesitation to say I am mainframe centric. That statement, alone and without context, will scare many people away from me as an IT consultant. One of the things I learned very early in my career is that Security of infrastructure is about People, Process and Technology. While the mainframe may be considered the most secure platform, technologically, my forensic experience at a variety of customers proves that poorly trained people and bad processes were the weakest links to security. But more important, much of that “poor security” happened at the end user device – formerly a PC, but now, including Smart devices, such as phones and tablets. If those devices aren’t secured with passwords and enterprise data residing on them isn’t encrypted, then they become the weakest link. And if the user of the device saves their userid and password in their browser so they can reconnect quickly, well so can the bad guy that steals their device and now, the bad guy has unfettered access to those “more secure” systems that execute transactions or provide data access on behalf of the end user that lost their device. I’ve spent over ten years looking at how back end systems can make the front end devices more secure. So I guess I am Security centric, as well. I’m also web, mobile and application development centric.

Most Application Developers are PC Centric.

If you started out as a mainframe programmer, you probably signed on to the mainframe with a 3270 emulator and used panel driven or command line driven tools to edit files and submit jobs to compile and execute the programs you created. The IT capacity that was used to do this type of development drove up the cost of operating the production mainframes.

The advent of the Personal Computer changed all that. Windows and Linux desktops provide graphic user interfaces. Fourth generation tools will help you graphically design the logic of an application and generate the source code in a variety of different programming languages that best suit the operations environment that you might run the program. With open system interfaces and common programming languages, one development tool might create code to run on dozens of operating systems and hardware architectures. These are the type of tools used to build most middleware that is sold to run across “your favorite” operating system.

Well, that hybrid development environment didn’t end up as simple as that. Tiers of deployment platforms were created. If it was developed on a PC, then the first choice for a deployment platform was typically a PC server. Other platforms, like UNIX and the mainframe, were considered primarily as production platforms. They didn’t distinguish very well or price differently for developers. As a result, it became unaffordable to develop for a mainframe because the development group or a new Middleware vendor, couldn’t afford a mainframe or UNIX server to test their code, so again, by default, most new applications were targeted to PC servers.

Most Web Servers are PC Centric
Most Analytic Servers are PC Centric

Need I go on? A mantra for the client/server computing era was Move the Data to the Application. This led to copies of data everywhere, but also led to theft, loss, data breaches and server sprawl. Virtualization of server operating systems has helped to reduce server sprawl, but security remains complex. Business resilience, environmental needs (floor space, energy, cooling) and labor costs remain highly complex as well.

I said earlier that I am mainframe centric. But I can also say, unequivocally, that the mainframe can NEVER solve all of your business problems by itself. Why is that? Because it is blind and deaf. The 3270 terminal and punch card are long gone as input output devices. The modern mainframe requires a graphically enabled front end device, such as a Point of Sale device, ATM, PC or Smart Device. It still requires communications but now it leverages TCP/IP instead of SNA. So any business leveraging a mainframe is now a multi system business. Even the zEnterprise, with its introduction of the System z Bladecenter EXtension can’t solve all of a business’ problems because it doesn’t handle virtual desktop infrastructure nor manage the deployment of end user devices.

So let’s go back to solving business problems. We don’t need to discuss server types, but we can make some statements that should prove true, irrespective of server deployment model.

Share data – the fewer copies of data, the easier to manage security and resilience. Sharing data for read/write access (transaction processing) along with read only access (Query and Analytics) will enable a combination of workflows that include real time analytics (fraud detection, co-selling) in a basic transaction.
Move applications to data – copying applications is far easier and less time consuming, in addition to more secure and resilient, than moving data. Virtualization technologies enable a simple way to bring applications and data together in the same infrastructure and improve latency and simplify business resilience.
Look for tortured data flows – there never will be a single copy of data, as there should be, at minimum, backup copies and disaster recovery copies. But if you can reduce the number of data moves, leveraging direct access to data, instead of file transfers or unload/reload workflows, a business can dramatically reduce operational complexity.
The fewer parts (servers and data) the better – there will be less environmental costs, software license charges and reduction in complexity for security, capacity management and business resilience management.
Use stateless devices/applications for end user connections – the end user wants direct access to data and transactions, but the less data stored on the end users’ device, the better. Cookies should be the limit of context stored on an end user device. That way, if the device is lost or stolen, no corporate data is lost. It will be stored centrally. This can be true of thin client computers as well as web access to a transaction processing environment.
Never give developers direct access or copies of Production data – Development systems are generally not production systems. There is no logging, limited or no security and rarely an audit of critical data. This is the simplest target for a hacker or insider to attack in order to gain access to personally identifiable information or corporate jewels. Developers should have cleansed data through anonymization tools or other creations to ensure that the production environment remains protected.
Measure availability end to end. I’ve seen server shops (of all types) claim four or five nines of availability for their infrastructure. That’s a nice internal measurement. If the end user, a consumer, is trying to access their data on a back end server and the security server, the web proxy server, a router or some other networking infrastructure is down then the business is down to them. Availability should be a business target not a server only target.
Identify the security of users from end to end. When looking at transaction logs, a business should be able to see the identify of the individual that initiated a request. If the logs only identify the id of a down stream server, then additional logs and correlation is required from more expensive products to identify the end user. The more misdirection the greater the risk to security and data theft. Ensure that the originating user is easily identifiable at each step of the business workflow.
Industry standard benchmarks are irrelevant to business workflows. They may help distinguish one component alternative from another. But since we’ve already determined that a hybrid environment will be required for production purposes, there are very few, if any, benchmarks that provide a true end to end workflow that mimics multiple business operations. Much like lawyers, benchmarks can offer guidance and may explain risks, but they are not the decision makers. The business application owner must weigh the total cost of operations and the incremental costs of maintenance for the entire business operations vs. just the component parts identified by a benchmark.

I first published “Porell’s Pointers” using Lotus Freelance for OS/2 circa 1990. The pointers covered then were server agnostic and remain true today. Sure, they were mainframe centric then and could be considered so today. But they make sense even if a UNIX system is the largest in your organization. They make sense if a PC server is the largest in your organization.

Published circa 1990 via Freelance Graphics for OS/2

If you can follow these suggestions, but also think about the scale of operations needed for your business, consider the mainframe as a core component of your end to end workflow. Unlike most other servers, it excels at large scale, managing service level agreements (SLAs) for read/write and read only data, providing cross platform security infrastructure and managing availability and business resilience at an application level. The combination of z/OS, z/VM with Linux for System z, along with PC servers and desktop or Smart Devices will be a winning combination to satisfy the majority of your business problems. But once you look at a hybrid solution, then security, availability, disaster recovery, application development, analytics, capacity management and backup and archive should be cross platform or hybrid solutions as well. A wealth of middleware exists that can operate across platforms. This can be the beginning of a new operations model built on business problems rather than server specific domains. Damn the Server Fiefdoms! Full speed ahead with an organization that collaborates for shared business success. Happy programming!

	IBM acquiring Red Ha… on Porting an Enterprise App to S…
	Jim Porell on Porting an Enterprise App to S…
	Garth Godfrey on Porting an Enterprise App to S…
	Porting an Enterpris… on Porting an Enterprise App to S…
	Porting an Enterpris… on Porting an Enterprise App to S…

Jim Porell and James Porell Consulting

Tag Archives: availability

Modern Data Usage Patterns – a case for Data consolidation

Observations about the Unintended Consequences of working with the IBM Mainframe

The Basics

Application Overview

Development Environment

Open Source and Operating System Dependencies

Rebasing the code and changing Development environments

Share this:

Vendor Development Team

Linux Community Development System for z

Scalable Virtualization

Open Source Access

Good people make life easier

Share this:

Linux is not Linux

Supported Platforms

Tolerated Platform

Unsupported Platforms

Docker containers are not portable across hardware architectures

Docker Stability

Ultimately, I have a wishlist for the Open Source Community on z:

Share this:

Java and Analytics run better on z/OS

Linux on z as a private cloud has more value than a public cloud

Docker inside z/OS? That would simplify things!

Savings and Operational Strengths

Bibliography

Share this:

Patterns

Where will the skills come from?

Adopt New Technologies and dump the old?

Past Technology Evolution Examples

Learn from the Fortune 500 – the good:

Learn from the government – the good:

Learn from the government – the bad:

Prescription for change

Modernization of the development environment

Language modernization:

Chip Speed

Risk and Fraud analytics

Collaboration is necessary to move forward:

Summary

Share this:

1. Digital Security

2. Virtual Personal Assistant

3. Smart Workspace

4. Software-Defined Anything (SDx)

5. Affective Computing

6. Neurobusiness

7. Prescriptive Analytics

8. Data Science

9. Smart Advisors

10. Speech-to-Speech Translation

11. Internet of Things (IoT)

12. Natural-Language Question Answering

13. Complex Event Processing (CEP)

14. Big Data

15. In-Memory Database Management Systems

16. Content Analytics

17. Hybrid Cloud Computing

18. Machine-to-Machine (M2M) Communication Services

19. Cloud Computing

20. Gesture Control

21. In-Memory Analytics

22. Activity Streams

23. Speech Recognition

The Evolution of the Mainframe.

Database Processing.

Evolution of new technology

Same code. Different Container. Different Operations Model

What’s your cluster look like?

System Integrity

Multi-tenancy

Summary

Share this:

Share this:

The mainframe is a dinosaur. The mainframe is dead

The long road toward making the mainframe relevant to new workloads

Dassault never looks back. Goodbye mainframe

SAP tells IBM what’s wrong with the mainframe

Network Performance and System Integrity

Data Character Translation

Most Web Servers are PC Centric
Most Analytic Servers are PC Centric