Tag Archives: system integrity

Hybrid Computing

Porting an Enterprise App to System z – my experience. Part 1 of 4: The Basics

4 Comments

At the end of 2016 and lasting a few months into 2017, I completed a proof of concept port of a large Enterprise Application that had been running on the Amazon Web Service Cloud to Linux for System z. This was a Docker based application written in Java…so of course, it would be trivial to port. WRONG. While the application is in Java, it called many pieces of open source code. Much of that code hadn’t been ported to System z yet or wasn’t widely adopted. What I thought was a very simple exercise turned into a six month effort.

What I’d like to do, via a series of blog entries, is share my experience in the hope this might help some other organization decide to do a similar porting task. While I’ve been working with mainframes for decades, this was my first Linux porting experience. So I’ll be describing how this experience helped me to Master the Mainframe, though that title seems reserved for university students.

This could be a book, but by breaking it up, it might be easier to understand.

  1. The Basics: High level overview of the application, the development environment, the system set up required to begin the porting exercise and the scope of the port.
  2. The Good: The people who assisted and taught me, the things that ported easily and the simplicity of getting started via the Linux Community Developers system.
  3. The Bad: the new open source for System z, the modifications necessary to open source to run on z, the debug experience and the time necessary to complete the porting process.
  4. The Future and Value. Regardless of the bad experience, there is a great business value in getting these types of Enterprise Apps on System z.

The Basics

This entry is more about the basic desktop development environment and targeted production on x86 based cloud servers. This is the traditional development environment and primary target of the applications. I needed to fit in and work with this environment before I could ever consider doing the unique activities necessary for success on Linux for System z.

Application Overview

Because of the proprietary nature of the application and intellectual property, I’m not going to name the vendor or application. This overview of the workflow is simplistic, at best, so as to not give away any trade secrets. The vendor is an early start up with an application to handle biometric authentication in a marvelous way. This application has a callable interface to start a request and then, using cloud based services, does some communication with the end-user, does some analytics based on a number of system defined characteristics, logs a number of things for diagnostics, audits and future analytics, provides a go/no-go decision back to the original caller and has a number of applications and user interface applications to manage the cloud deployment. Finally, they have an enormous test suite to emulate and automate the entire end to end workflow.

Development Environment

This vendor was doing all of their development for the x86 platform and originally with any Linux version supported by Amazon Web Services. This included Centos/Red Hat versions. Their first development environment used Maven tooling and pom.xml scripts that targeted deployment into Docker containers. They used Github capabilities to clone and manage the source code libraries within their business.

The first major effort was for me to establish a development environment on my computer and prove that I could work with and build a workable x86 version of the code. My computer of choice was a MacBook Pro 2010 model running the latest MacOS at the time. First thing to do was turn my MacOS into a real developers machine. I installed xcode, Atom, SourceTree, Filezilla and Docker which enabled me to look like a Linux system, edit source files intelligently, manage access to the source files, facilitate cloning of source and execute the code. There were other local variant software that I needed to install using a script that was provided to me. I love the Mac, as did the vendor, who’s entire team used it, so that was really helpful. I then needed a VPN into their system and I was off and running. I used this set up for about two months. One thing I learned, painfully, that the 2010 Mac was SLOOOOOWWWW. What would take 15 minutes to do for them might take me over an hour. So I decided to upgrade to the MacBook Pro Touch Bar quad-core 16GB memory laptop. Now my work completed faster than their 15 minutes, which was a blessing. I can’t stress enough the value of a good starting point on the desktop or laptop for this type of development! It was life changing to me.

Open Source and Operating System Dependencies

The first version of the vendor code used Centos/Red Hat as the target deployment environment. This code runs over 50 Docker containers. Each container is intended to be as small, memory wise, as possible, so it is scalable in a largely virtualized environment. As mentioned earlier, they also used Maven and pom.xml scripts to do their container builds. Each container had a script that would gather necessary pre-requisite open source parts, their Java code and then do the build so there was an executable container. Naming conventions, versioning and more were part of these Maven scripts. 90% of the open source code used was available in a binary form as either an RPG, ZIP or TAR file. Those binaries were either copied into the vendor’s library system or accessed via a URL and dynamically downloaded from the internet during the build process. I’ll get into the System z ramifications of this in the Good and Bad blog entries.

This is the development environment I began my first phase of the port. The prototype I was building was only for a functional test to prove the code could work. We intended to accomplish our test goal with only 40 of the 50 containers being ported. We completed what we thought was a good test level of code after a few weeks of my porting. But then we identified some critical test containers were missing. Unfortunately, the vendor didn’t use the same library management rigor for their test suite and I was going to have to re-base my code.

Rebasing the code and changing Development environments

Unfortunately, that was the tip of the iceberg in changes. I mentioned this was a startup vendor. They had two very large customers that were testing the code when I started. They realized they had a scaling problem, early on. They also realized they had some development inefficiencies. When you get a RedHat, SUSE or Ubuntu distribution, there is a lot of software in the package, like getting the z/OS operating system, MacOS or Windows. As such, the kernel of the large distribution Linux systems can start at 250MB and easily be over 750 MB’s. When you add 100’s of virtualized containers, each having that size as the basic footprint, the overall system runs out of memory pretty quickly. However, if the kernel can start at 18MB and run about 50MB, then greater scale is possible. As development of this application began, the Alpine Linux distribution began and it met the small size requirement. The vendor began to rebase all of their test code and as much of the open source code as they could on Alpine to take advantage of this reduced memory benefit. That was and is an excellent business decision on their part.

Maven is a fairly complex environment for building docker containers. It works. Both the vendor and I proved that it could work. However, in addition to open source code, there are now open docker containers that can be leveraged, as is, to be included in place of an open source binary. However, in order to do that with Maven, the Docker definition files of these open containers must be cut and paste and then modified as part of the Maven script syntax. And each time the container definition changes in the open source world, the Maven scripts need to be hand modified. So the vendor dropped Maven as the base for their container build environment and switched to using Docker build definitions directly. Again, I applaud the vendor for doing this. It simplified the development environment, it gave them access to additional open source code repositories and made everything easier to manage.

The unintended consequences of the vendor’s change from Maven to pure Docker and Centos/RedHat to Alpine was I had to start all over on the port. I’m going to save the details of that for the Good and Bad statements as they are directly applicable to System z.

As far as Linux for x86 cloud environments, this vendor has a world-class development environment, working to create the most reliable, secure and efficient application possible. Ultimately, those attributes must apply to System z deployment as well. I’ll be covering that status in the other blog entires.

Hybrid Computing

Porting an Enterprise App to System z – my experience. Part 2 of 4: The Good

Leave a comment

I provided a simplistic overview of what I intended to port to Linux for System z in Part 1. The original application was built for x86 systems. As such, all binaries are built to run on x86 systems. The Docker containers that these applications run in are x86 binaries as well. So my job was to create the Linux for System z (aka S390X) binaries, with as little change as possible.

I also mentioned that this was a start up vendor with whom I was working. I had done some business work to show them the value of porting the application to System z, but they were neither skilled in, nor able to afford their own System z. So I gave them the challenge to let me prove to them this could be successful and they took me up on it and agreed to work with me.

Vendor Development Team

While a small development organization, they still had over 25 very proficient programmers and testers. I was extremely fortunate to have their lead developer as my mentor. He and I would meet at the same time, for an hour every day to check on progress, educate me or diagnose any problems I might have so that I could make progress for the next day. Most important is he was learning about the mainframe and intrigued by the possibility of business success as I was, so it was a great experience for both of us. I greatly appreciate the time and effort he put in to make this a success.

Linux Community Development System for z

Where do you find a mainframe? You ask the Community Development team. Eva Yan at IBM was instrumental in approving the vendor and I to get access to Docker containers on the mainframe. Cindy Lee at IBM was fantastic, with her team, to help show me where all the open source for z was available in the community and Martha McConaghy at Marist College, the host for vendor access to the LCDS was terrific in helping me to keep the system running.

Docker is a great place to work with portable code. My development environment was an x86 Docker container environment that pointed to the S390X Docker on the LCDS system as the target deployment environment. I’m not going to spend time giving you the details on the set up, but suffice to say it all works well.

Scalable Virtualization

I didn’t mention before that the vendor is on a different continent. So imagine from my laptop, a VPN to the vendor’s libraries where some code is downloaded, merged with code on my desktop, Docker on my desktop puts all the parts together, ships it securely to the Docker on the mainframe image, does the build and sends results back to me. So if this process took 10-15 minutes to do on my laptop, suffice to say, when you add up the networks and bulk distribution of code between systems and do the build, it’s going to take more time than a single system. Doing a single container build, for the first time, was never correct. My mantra, for years, has been “Next time for sure!”. I’d fix what needed fixing, get a little farther the next time, repeat the mantra and try again, until finally, I’d get a successful build. The time or performance isn’t a problem when building a single container. It’s when you build 40-50 containers at once, or as I liked to call it “The Big Bang”. Then it was hours to do the build on the mainframe, instead of an hour on x86. You’d think that was the bad, right? It was good, because a call to Eva, requesting some more memory and processors and I moved to a very competitive deployment environment. For just like my MacBook 2010, which was under configured for this scale of development, the initial Linux system I was given was an under configured virtual machine. With a simple config change, within moments of my request, and literally no down time, I was up on a larger Linux image, due to the magic and wonders of the underlying scalable z/VM server image.

Open Source Access

The LCDS virtual images came with RedHat kernel as the base, with some optional software included, but that was all. I need several dozen pieces of open source software to add to my environment to build my S390X binaries. Again, I don’t want to spend the money to buy a supported Linux distro for this Proof of Concept. I’m directed to Sine Nomine Associates, and in particular to Neale Ferguson. He could not have been a better ally in this effort. First and foremost, he pointed to libraries on their servers where I could retrieve many of the binaries that were necessary. It was such a relief to find many of the rpm’s I needed on their website. As mentioned earlier, I was a newbie to this kind of porting. He spent considerable time mentoring me on both basic Linux and System z specifics to keep me moving along. As important, Neale was on the Docker band wagon. He’d begun building docker containers with specific functionality. I was able to take several of his containers and imbed them into the containers I was building to simplify my deployment.

The Linux Community also has Github repositories of System z ready open source code. I bookmarked those pages and visited them often. I’m pointing links in a Bibliography in Part 4.

The real dilemma came when the vendor switched from Centos to Alpine as the base Linux kernel. Alpine was so new in late 2016, early 2017. While both are Linux derivatives, the syntax of packaging applications is different. As such, Docker builds for Centos are different from Alpine. Because I was doing a proof of concept, it really didn’t matter whether I used Centos or Alpine. However, the longer my porting took, the faster the vendor was converting their code to Alpine, so now, I would have to make “throw away” changes to support Centos.

Worse than that, there was only one person even trying Alpine on the mainframe and that was “some college kid” as a research project. How could I build an enterprise application on a system that one unpaid person was supporting? That person was Tuan Hoang and I am indebted to him. He was a Marist College student. I began contacting him late in 2016. While he had the kernel ported, there were very few packages for Alpine ported to S390X. He was quickly up to the task. I gave him a list of high priority packages. Each night, I’d get an update of what he completed. Each day, I’d build some more containers off his evening’s work. It got to the point that only third-party open source packages were not done by him. This really got my development effort going. But the best news of all was at the end of my project. Tuan had worked so hard to get his “prototype” of Alpine for System z going that the Alpine community accepted S390X as a primary target platform. All Alpine packages would be available on S390X, simultaneously to their deployment on other hardware architectures. It was painful, but it was wonderful at the same time.

Good people make life easier

What I found throughout this porting effort is there is a wonderful community of people dedicated to the support and value of System z. They were very accommodating and helped reduce my efforts greatly.

Hybrid Computing

Porting an Enterprise App to System z – my experience. Part 3 of 4: The Bad

Leave a comment

As I’ve explained in Part 1 Basics and Part 2 Good, I did a proof of concept port of an Enterprise Application from Amazon Web Services on x86 to Linux on System z in 2017. The good news was I got to the point I needed to, the bad news was it was more than difficult to get there.

Linux is not Linux

Open Source is open source…available to anyone. The story goes that Linux is Linux. Close, but not quite. Unfortunately, architectural chip bits (Big Endian vs Little Endian) is one of many differences and there is code that needs to change to handle these differences. There are also supported platforms, “tolerated platforms” and unsupported platforms. This is the problem with Linux on System z. The marketing hype is that all of Linux is supported on z. The reality is somewhat different. Not necessarily insurmountable, but you better know what you are getting into.

Supported Platforms

When Linux on z is a supported platform, then the packages for System z are supported in binary format, such as an RPM file for Centos/RedHat or an APK file for Alpine. This is the best case and makes development of S390X on par with other platforms like x86 and ARM.

Tolerated Platform

In this case, the code may work on S390X, but it’s a source code build. You can find instructions on Github for S390X as to how to modify the code to get it to work on the platform. But if you want to use that code, it could take a long time to

  1. Do all the things necessary to manually modify the code
  2.  execute the code to create a binary.

Let me use an example. Couchbase is the non-SQL database preferred by the vendor I worked with. Someone within IBM is maintaining a script on Github to help others leverage a particular release of Couchbase. Since Couchbase is constantly coming up with new versions, those edits need to be constantly updated. I would have preferred a binary version of the code, but IBM doesn’t do binaries…They only do source. And in order to make Couchbase work, there are pre-requisite source modifications necessary to Go, Python, cmake, Erlang, flatbuffers, ICU, jemalloc,  and v8 javascript. Manually doing all that is necessary takes a few hours. I was fortunate to take all of these changes and build a docker script that was several hundred lines long to automate the build of Couchbase by doing all this work. When I ran this container build, it took over an hour to complete. I had to do this many times before I got the automation script to work properly. And that automation is only good until the next release comes out. In comparison, with an x86 rpm, this takes a couple of minutes and the Docker script is about 15 lines long. In the end, I got what I needed, but the level of effort to get there was tremendous. I also mentioned container memory size in Part 1. This Couchbase container on z was over 1 GB memory. This put a tremendous strain on Docker and we found a few bugs as a result. The size was a combination of Couchbase and all the prerequisite code  to build Couchbase. So I had to modify the Docker build to delete all the prerequisite code which included source, binaries and documentation. This got the container down to a more reasonable execution size.

BTW, when I complained to IBM leadership about the lack of support for Couchbase, they suggested I use a different, easier product that was available on z. Since I was porting and not a true developer, this was not a possibility for me. I had begun negotiations with Couchbase toward this goal, but stopped working on it when the prototype ended.

Unsupported Platforms

There were two cases where neither the open source community nor the Linux on z community had guidance on how to get a particular open source program on the mainframe. In those two cases, I was able to get through the code, successfully and get a binary for System z. The good news was it was pretty simple to do. I was quite fortunate. If it hadn’t been easy, this could have ended the project earlier than I had hoped.

Docker containers are not portable across hardware architectures

I’ve seen some hype that once you get it in Docker, it’s portable to any Docker. I’ve heard a few mainframe customers believe any Docker container can run on System z. I’ve also seen articles in IBM sponsored magazines that purport this to be true. This is a combination of marketing hype and misunderstanding. It all depends on the container architecture/binary and source code. Typically, a container binary for a particular architecture, such as x86, should run in a Docker container on any x86 platform, even if it’s a different operating system running Docker. For example, Docker running on x86 version of RedHat 7.3 could be running containers with RedHat, SUSE, Alpine, Ubuntu, etc, as long as they were built for x86. Similarly, I ran Docker on a RedHat 7.3 image for Linux on System z, and had containers with Centos and Alpine running with binaries for S390X.

The only containers with source code that were portable were built exclusively with interpretive languages, such as Java or Python. Those could be portable across hardware architectures. Many of the test cases used by this vendor fit into that category. However, as soon as one of those interpretative languages makes a call to open source code middleware (e.g. Couchbase), then the container is no longer portable across architectures because the middleware is not supported across architectures.

Docker Stability

When I started this project, Docker on z was pretty new. Once in a while, it would have issues. Only a couple of times did it require Marist College to restart my z/VM guest. The other times, it would automatically recycle itself and get running again. I believe it’s improved  since we began the port effort until now, but it’s been a few months since I tried it. I’ve heard from others, though, that the experience is better. During our Big Bang builds, we would peg each of our System z processors at 100% busy for a few hours. The fact that it would stay up and continue processing is a testament to the reliability of those large code tests.

Ultimately, I have a wishlist for the Open Source Community on z:

  1. Where source code changes are necessary, such as with Couchbase described earlier, supply a Docker build file to automate it for anyone that wants to do the build. It would be so much faster.
  2. Continue to lobby third-party open source middleware providers to support system z. In many cases, it takes a vendor, such as I was working with, to create that business case jointly to get it done, but doing that will lead to more usage on the platform. If you build it, they will come.
  3. Create more binary packages instead of source code update files. It greatly reduces the development time necessary for z unique porting. The more extra work necessary to support z, the less likely the x86 people  will move there.

The net of all this bad is the initial effort to support the mainframe is longer than it would be on x86. However, if you have the patience to get to Part 4: The Future and Value, you’ll find that you should be rewarded for the effort.

Hybrid Computing

Porting an Enterprise App to System z – my experience. Part 4 of 4: The Value and Future

2 Comments

In Part 1 The Basics, Part 2 The Good and Part 3 The Bad, I’ve explained I did a proof of concept port of an Enterprise Application from Amazon Web Services on x86 to Linux on System z in 2017. The good news was I got to the point I needed to, the bad news was it was more than difficult to get there. But why did I go there in the first place?

The vendor for the Enterprise application was targeting the Financial Services industry for their initial deployments. This is the primary customer for IBM System z. Their beta customer is running z/OS transaction processing via CICS, but wants to authenticate customers using this vendor’s product running on Amazon Web Services. In order for CICS to call the AWS Cloud, it has to launch Websphere on z/OS to call the vendor’s  service on AWS. The vendor’s application has to do it’s task of authenticating users and get all the way back to CICS in less than 18 seconds so the transaction doesn’t time out. It’s a really powerful use of the vendor’s application and valuable to both the consumer and financial institution to avoid potential fraud or cybersecurity scams.

Java and Analytics run better on z/OS

I was told this vendor wrote all their code in Java, so I immediately began a plan to get this running within z/OS, since Java runs so well there, especially on the z14 systems. I also knew that in the time allotted to run on AWS for those 18 seconds, only three biometric/analytic tests could be completed on behalf of the consumer. I hypothesized that if the vendor app ran within z/OS perhaps up to ten analytic tests could be completed using the outstanding analytics and Java performance. However, once I learned of the number of open source middleware programs required and the complexity of porting them to z/OS, I went to Linux on System z as the target port.

Linux on z as a private cloud has more value than a public cloud

Using RDMA as the memory based communication between z/OS and Linux LPARs, I know it will take a bit more time than running inside z/OS, but much less time than going to a public cloud, so I hypothesized that eight analytics tests could be done instead of the three on AWS. And regardless of z/OS or Linux on z implementation, the vendor agreed that the software price would be the same as AWS. The net is, z would have additional analytic value, and given it’s hardware and software integrity and reliability, it would offer better security and business resilience than any public cloud provider.

So that’s what I set out to prove. Sadly, I got so close and the vendor changed their mind on their business strategy. They received a significant new round of venture capital investment, signed up several new financial firms to try their code and they decided to stick to their current cloud plan and stay off the mainframe, for now.

I still believe that my hypotheses as to the performance and value were correct. But the activity ended just before I was able to prove that. However, the exercise did confirm the possibility of getting the product on the mainframe successfully.

Docker inside z/OS? That would simplify things!

But what else is possible? I said in Part 3 that Docker containers are not portable across architectures. However, they are portable within the same architecture. There are some prototypes underway for Docker to run within z/OS. Given the way Docker works on other platforms, it would infer than any Linux on z containers could run unmodified within z/OS. If Docker for z/OS were to run on a zIIP processor, there would be no software license hits for z/OS. If that all comes to pass, that could lead to significant transaction and analytic value within z/OS and greatly simplify the system management requirements for these types of hybrid workloads, while improving the overall security, resilience and performance and reducing the operational costs. I would hope that a public announcement of this capability is not too far in the future.

Savings and Operational Strengths

That, my IT friends is a win for everyone. Any of the bad associated with a slightly more complex development environment can quickly be eradicated with a greatly reduced operational expense that has greater operational benefits than any alternative architectures might try to demonstrate. This type of workload makes for a very compelling end to end benchmark comparison as well. So while I didn’t succeed in getting the enterprise application to market, that was because of a business decision rather than a technological impediment. And the business decision was tactical, based on their new financials.

I learned a lot and documented many of the short cuts I took and set up required to make this development effort possible. I’m happy to share the experience if you’d like to undertake your own development effort. While I thought the end of the project was a failure, it’s unintended consequence, with the efforts of the great Linux for z community identified in Part 2, is that this will be easier porting for everyone that follows.

Bibliography

LinuxONE and Linux on z Systems Open-source Team

LinuxONE Developers Works

Neale Ferguson’s pre-built Docker containers for z

GitHub repository to S390X open source scripts  From this page, search for the package you are interested.

 

Hybrid Computing

Mainframe Security – How good is it?

Leave a comment

Two things about security that have been true for a long time:

  1. The Mainframe is the most secure platform in the industry
  2. Security is about People, Process and Technology

These are not mutually exclusive concepts. What’s important to realize, though, is the mainframe shouldn’t get a “Pass” on security processes because of its reputation. Unfortunately, I have encountered some really poor security management practices associated with mainframes at a wide variety of customers. These poor practices have put those businesses at risk. Mainframe technology and architecture can inhibit a number of security issues that other platforms regularly encounter, such as buffer overflows that enable viruses and Trojan Horses. However, poor management of data access, protection and audit can lead to data loss, theft and network attacks.

The Mainframe is NOT Hacker proof

There, I’ve said it and I’ve said it before. Mainframe systems have been hacked. There is one obscure case where some very old open source code was running on a mainframe. That open source code had been successfully hacked on other platforms. The attackers used a similar technique to get “inside” a business. Network attacks that drive a denial of service have also been successfully initiated. Both of these types of attacks were clearly avoidable. The worst attacks have come from insiders. Sometimes by accident, but unfortunately, sometimes on purpose. Not unlike Edward Snowden and Wikileaks, insiders have released confidential information stored on mainframes. In each of these cases, better security practices and the use of additional products and monitoring could have inhibited these data thefts.

Who is responsible for Mainframe Security?

Another problem that I’ve seen is lack of diligence by a business over the people managing the mainframe. The worst case scenario dealt with an outsourcer. The outsourcer mistakenly assumed that the mainframe was “hacker proof”. The owning business assumed the outsourcer knew what they were doing and didn’t audit the security of the systems nor the outsourcers running the systems. It turns out, the outsourcer wasn’t auditing the security either. In this instance, network ports were left open that enabled attackers a way into the system. Data sent over the network wasn’t encrypted, allowing attackers to sniff for critical data. Perhaps worst of all, many of the systems programmers had operational access to modify all system datasets without going through change management.

It doesn’t mean that anything negative happened at this customer. But it certainly could have happened. And even within that particular outsourcer’s business, they had other customers that were properly protected. This seemed to be a local anomaly. The lesson learned, however, is that the owning business should own audit responsibilities and the analytics associated with their security operations and data protection. Simple tools could be deployed and routinely run to “health check” their operation. Either the outsourcer or the business can run those tools, but the business should check the results as a normal part of grading their outsourcer.

“Knowledgeable” People may be your worst enemy and largest risk

Social engineering is a terrific mechanism to get access to restricted data. I’m not going to give examples, other than to say that asking the right questions of the right people can result in acquiring data or privileges that someone isn’t supposed to have. Again, the human element at work in circumventing security. One of my favorite “social engineering” episodes was visiting the security director at a large bank. Physical security was tight. The team there asked if I could break into the building. My normal response is never on the first time, but usually on a second pass, it is possible. As I left the cold building in December, I realized I left my jacket in the Director’s office. The security guard told me to go back up to the third floor unescorted. I was in the Director’s desk chair when he returned from the rest room. Isolated incident? Unfortunately not. That’s just my favorite example, without the details.

Replicated data doesn’t mean that security control is replicated

It doesn’t matter what platform or server you are using for a database. Far too many businesses make a copy of production data so that the Quality Assurance team can do system modifications and stress test before making updates to a production system. Application developers might also have access to this production data to test the next iteration of a business application. QA and development may be outsourced to another business. That business could be in another country. Without audit or security management of the test and development copies of the data, there is the very large possibility of data theft or leakage.

The weakest link is the End user interface – PC’s, Smartphones, Tablets

Plenty is known about the security problems associated with end-user devices.  Management of those devices falls on the owner of the device. Where Bring Your Own Device or BYOD is allowed, then the management practices for the devices will differ by the number of devices. Unfortunately, users save the userid and passwords of back-end servers on these devices. As a result, any server that these devices access is at risk of mismanagement or spoofing of credentials of the end-user if that device is stolen or hacked.

There is hope. Collaborative security should be the norm.

Earlier posts of mine discuss collaborative and hybrid computing. This is as important with security as it is for business resilience, storage management and application development. By looking across the IT infrastructure, a business can identify risk more clearly than a business that has fiefdoms protecting their smaller domains. Analytics, audits, identity management and data protection done across the IT infrastructure will help a business reduce risk and save on overall costs.

Don’t let security by obscurity result in the unintended consequence of data loss. Stay vigil. Keep an eye on your systems, your system administrators and your users.

Hybrid Computing, Unix System Services

If you started with computers in the 1990’s and hated mainframes – you were right

4 Comments

Wow, I said it. It’s hard to admit for some of my peers, but looking back, there is a lot of truth to that sentiment. I’m not talking about traditional transaction processing used for insurance claims, point of sale, ATM’s, travel reservations, etc. That’s where money is being made and continues to be made, with many businesses leveraging mainframes as their System of Record.

No, that’s meant for folks that were and are  interested in client/server deployments. Folks that wanted to use modern application development tools. Folks that wanted to incorporate multi-media and streaming into their workflows. These might be considered the System of Engagement today. Oh my goodness, the mainframe was not at all prepared to handle that work. Decisions were also made for departmental computers. Can you imagine a mainframe of that era in a closet? A closet? Where was the chilled water, the raised floor, the humongous air movers?

IBM_Main_9000

It was a traumatic change within IBM to begin the modernization of the platform and 1. keep it relevant to where it was having its “legacy” value (the prime objective) and 1A. make it relevant to capture new workloads. This was something tolerated, but not completely embraced at the IBM executive levels.  Ignoring the success of “desktop computing” and new business opportunities, I always felt that the mainframe sales mantra was kind of the opposite of the famous Star Trek line: To boldly go where we have already sold before. So the real goal appeared to be to keep our current mainframe customers happy.

The mainframe is a dinosaur. The mainframe is dead

There were voices, everywhere, saying the mainframe was dead, it’s a dinosaur. It chilled the hearts and minds of senior IBM executives too. There was a “new breed” of executive that wanted to look and grow new business lines, like the PC Server and Power Systems. Where did their funding come from? The profit of the mainframe. But as a result, mainframe R&D budgets were challenged. And even within the mainframe, as growth opportunities were considered, the development budget was spread even thinner across a wider variety of efforts. Some believed it was a cash cow, from which new opportunities should be funded. And then there were the mainframe believers that had to fight “the new status quo” to maintain their budgets. I could go on and on about these political battles inside IBM. There’s some TMZ quality stuff, but not what I want to discuss here.

Now, if you don’t want to read the story about what changes were made and how they were made, you can skip to the Summary of how bad it was and how good it has become.

The long road toward making the mainframe relevant to new workloads

Instead, I’d like to tell you about the changes that were made to make the platform relevant to new business opportunities. In the process, the positioning of the mainframe changed. It’s still a terrific System of Record, but now, unlike the 1990’s, it can be a viable System of Engagement for many functions, but not all. Because if I said all functions, I’d be lying every time I mentioned hybrid computing!

Let’s start with a big mainframe app that made the decision to go “all distributed” and never looked back. That hadas much to do with money as it did with technology.

Dassault never looks back. Goodbye mainframe

Dassault System’s CATIA application – a CAD/CAM – engineering modeling/rendering application.  They were charging $8K a user for the MVS version. Each user had a specially built 3270 Graphics Adaptor for model renderings on a “green screen”.2250Graphics

There were so many instructions required to do the renderings that each terminal had it’s own hardware interface. At the time, there were only 4096 interfaces (called Unit Control Blocks – UCB) on the system. As more engineers created modern airplanes, the customers had to buy more systems, which were not cheap to acquire, nor operate. Ultimately, the z/OS operating system eliminated the constraint on the number of UCBs, which would have greatly helped CATIA customers, but had the added benefit of helping all customers when Parallel Sysplex was introduced later.

CATIA grew bored with the proprietary 3270 GA box as their graphics rendering device. They wanted color and simpler graphics.CATIA-14X-actuators-135x90

This is just what the UNIX community was demonstrating. So they created a new product, based on AIX. They thought the UNIX based version should charge about $10K a seat. Some cooler graphics were worth the price, but not too far off the current customer price. They did a market study. They found other UNIX based engineering programs were getting about $25K a seat  And those systems were selling well. What a profit margin! So they decided to charge $22K a seat, in order to undercut their competitors.  They also yelled from the rooftops “the mainframe is dead” in order to transition to the new model, but more importantly, the new revenue and profit stream. What was surprising to me was that Dassault was partially owned by IBM. My job was to talk to the “chief yeller” and convince them of the “new mainframe strategy” which was still vapor ware, from their perspective. All requests to stay on the mainframe fell on deaf ears, and rightfully so, in hindsight.

SAP tells IBM what’s wrong with the mainframe

A “big mainframe” app that transitioned to commodity systems was from SAP, which was mainly a bunch of ex-IBMers that had a really good idea. SAP R/2  was created with a mainframe back end application and database server which also drove the green screen terminal front ends.
SAP R/3, the next major version,  was a “client/server” app that was all about the end user experience (GUI) and commodity server deployment models. It didn’t run on the mainframe when it first came out. SAP was one of the voices bragging about the “death of the mainframe” in order to transition customers to their new product. It took years of negotiations and modernization of the mainframe infrastructure and pricing changes for SAP R/3 to “return” to the mainframe, but it was only a partial return….The Database, entirely, and then the data access methods and some of the applications. The presentation layer has expanded beyond the desktop into mobile and web services. The original application programming language remains the same, though it has been augmented with Java.

Let’s peel the onion and look into modernization specifically for SAP R/3. In my opinion, it was that application, or better stated, that company that forced IBM to change the mainframe. IBM wanted new workloads. Here was a big one that took business away…Let’s win it back.

Network Performance and System Integrity

First, client/server was based on the internet and local area networks. They were rapidly transitioning to TCP/IP hosted networks – an open network. IBM was still stuck on SNA, it’s proprietary network that was actually a de facto industry standard as most systems could interoperate with it. Today, SNA applications can run over TCP/IP….it’s about layering, but that’s too detailed. An implementation of TCP/IP for MVS is created. Check the box. Done. Well, wait a minute…how well does it perform, SAP asks. It takes about 186,000 instructions to send a small chunk of data out and receive the acknowledgement of successful delivery. By contrast, it takes AIX about 18,000 instructions to do the same. So MVS is about 10 times slower communicating with TCP/IP than other platforms. Well, no it’s not. The SAP R/3 application is very chatty. The application server makes a network connection to the database server on the mainframe. For each end user request that ends up making a database call, there are actually 26 send/receive pairs for each transaction. So the overall pathlength is 260 times worse than AIX for a single SAP transaction. At the mainframe hardware and software price levels, that isn’t going to sell well. I also made a joke of this at the time. What’s Lassie, RinTinTin, Benji and TCP/IP on MVS?

Three movie stars and a dog. Drum roll please. This was a bad situation. Unfortunately, this joke lasted several years. A lot of effort went into looking at alternative implementations.

Anytime there is a network, data must move from one system to another system. In a terminal environment, such as a 3270 screen update, it might only be 100 bytes of data get transmitted. But a video could be millions of bytes. IBM, from it’s outset, has been historically great at creating architectures. Some architectures have layers of functionality. Communications or networking architectures are famous for the seven layer diagram,

Osi-Layer-Modelwhere each layer has an architectural purpose. Application layer, transport layer, session and network layers being a subset. IBM has an image processing application originally developed for medical records called Imageplus. The performance of image processing (and later streaming) over the network, because of the rigidity and adherence to the layers of architecture by the original code base, was horrendous. The physical image was COPIED TWELVE times, once for each layer and a few times within the application, as it traveled through the operating system. Eventually, the networking software was modified so that it could copy the data from the application buffers directly to the transmission wires after a quick test of the applications data integrity (aka make sure there would be no buffer overflows or attempt by an application to send system or privileged data that they didn’t have authority or “need to know”.  This data integrity testing, by the way, is inherent throughout the mainframe operating systems and middleware).

So the Imageplus benchmark was also critical to understanding how to re-write the TCP/IP stack. If the system knew the SAP Application server was running on an AIX server that was channel attached to the mainframe, the new performance was 5,000 instructions. (Didn’t need to do some network error handling  nor routing and this saved instructions). If the Application server was attached via a router, then it was 10,000 instructions.  Hallelujah….we were better than the rest of the world…..

Data Character Translation

Not yet, says SAP. Those desktops, those application servers on Windows and UNIX/AIX? They like to read and present data in ASCII format. Your mainframe and  DB2? They like to process data in EBCDIC format. It takes seven instructions to translate each byte. So all that network gain that was just achieved is lost by data translation. As a result, effort is put into the path length of data translation. Eventually, it gets down to three instructions per character. With the network savings, this is roughly equivalent, end to end of a commodity system performance. With price/performance, it’s still not good enough. As a result, the mainframe hardware, operating system and DB2 middleware are changed to natively support unicode characters, wide characters (for character based languages) and many other code pages. There was, and is, no longer a penalty for code conversion. Finally, after several years of work, SAP R/3 sales of hybrid computing began.

New workload pricing

But the transition isn’t complete. The introduction of Linux on System z leads one to believe that the mainframe is a viable application server. Not so fast, says SAP.  IBM charges software by the MIPS or capacity of the ENTIRE machine.  So if a customer adds one processor to run Linux on a mainframe that has 9 existing processors, the Linux license charge would be for 10 processors worth of work vs. the single processor that is executing. I’m going to rapidly jump ahead, but specialty engines were created such that new workloads, such as Linux, JAVA on z/OS, distributed connections to z/OS databases, some z/OS system utilities and more are charged by the actual engine or capacity of the workload vs. the entire capacity of the machine. Finally, incremental workloads could be added to the platform at or near commodity prices for the equivalent work. But we’re still not done with this evolution in hybrid computing.

Stop copying data. Share memory instead

Avoiding data moves was critical to the success of the “new” TCP/IP within z/OS. So next, the same concept is applied WITHIN the mainframe. And this takes several iterations with continuing benefit.

In a Blade server or  rack mounted system, a special back plane or Top of Rack switch may be deployed so that communications within a ‘server box’ can go point to point, ala the mainframe channel to channel interface and avoid router overhead. Under z/VM with hundreds of Linux images running and then z/VM to z/OS within the same physical server, IBM created the hipersocket which uses dedicated hardware memory, instead of wires, to facilitate intra-physical server communications. The most recent announcement upgraded the hipersocket technology to leverage RDMA memory that can be shared between server images so that only a pointer to the data is transmitted.  Each system can have direct access and eliminate intra-server copying of the data.

Increased capacity and memory available for new workloads

All of the above mentioned activities result in fewer data moves within the operating systems, virtualization layers and hardware servers themselves. This avoids instructions, real memory and latency required to do data moves. This allows the processors and memory to be available for other workloads. This allows greater scale of the environment. And because the mainframe has been doing this for over 40 years as a balanced architecture across instructions, memory, networks and data, instruction pipe lining, many levels of memory caching and more, it is capable of putting, colloquially, 10 pounds of “stuff” in a 5 pound bag without fear of breaking. Said another way, there is no fear in running the system at 100% utilization for very long periods of time.

The “Legacy” workloads get modern as well

So that’s just a snippet of the technology that went into “modernizing” the mainframe for new workloads. There is significant other infrastructure, such as the Parallel Sysplex, Geographically Dispersed Parallel Sysplex (GDPS), evolution from BiPolar to CMOS technology, reductions in cooling, electricity and floor space, packaging of “spare” parts in the box for fail over/fault avoidance and on demand deployment for capacity upgrades and more that made the mainframe better for traditional workloads, and over time, those benefits have applied to the new workloads as well.

But then, how much does it cost?

Historically, the mainframe has been considered an expensive alternative to “commodity” platforms that are “just good enough”.  Much has been done to change the pricing structures.

Technology Dividends

Since the introduction of OS/390, IBM and other vendors have been providing tech dividends such that the price/mips has decreased regularly and with most new processor introductions.

Capacity Backup

The “z” in System z, stands for Zero down time. While not reality, it is a goal, both at the technical level and through software license charges. IBM introduced Capacity BackUP (CBU) pricing for backup or disaster recovery servers. A fraction of the production price is paid for the hardware server and no software license charges are paid. Many other vendors have accepted this model as well. Why? Because a business isn’t getting any productive work out of the backup servers. When production work moves over to the CBU server, then the software license charges transfer to that machine. A business may do several “fail overs” a year, to test recovery operations without incurring a license charge. When compared to “commodity” servers, this can be a tremendous savings. In those environments, multiple servers may be required and each of those will be charged a production license fee.

Development pricing

If a business wants to get more workloads, then it needs to cater to application developers. Those developers need a sandbox or low priced system to create that code. Unfortunately, software license charges being as they are, all MIPS on the mainframe were created equal and treated as production. Rational was a brand acquired by IBM. They had created some fantastic tools that worked across platforms and now included OS/390 and later z/OS. Those applications would require CICS, IMS, DB2 and other vendor middleware as the target production environment. However, IBM was charging the same price for the production licenses. Most other vendors, including Microsoft, Sun, Oracle and HP were giving away the run times when they sold their development tools. As a result, the mainframe was not targeted by most vendors as a viable deployment platform because the “cost of entry” was way too high.

IBM finally released a mainframe architecture that ran on PC and Power servers – the z Personal Developer Tool (zPDT) to vendors. This created a competitive cost for vendor developers to target the mainframe as a production platform. It was several years later that IBM decided to make this available to businesses or customers. Finally, there was a common tool set, Rational Developer and free run times and test environments that were priced competitively with “desktop” development tools. Better yet, a developer could “take a mainframe home” with them, as the zPDT could run on a Thinkpad laptop.

Parallel Sysplex pricing

Clustering computers is an important way to scale. IBM invented the Parallel Sysplex architecture that accomplished three things, with two being a huge differentiator from “commodity” server clustering.

  1. Load balancing work across the cluster
  2. Shared Lock management of database and file records across the cluster
  3. Shared cache of the database and file records across the cluster, allowing all cluster members to have direct access to the data.

The direct access of shared locks and data cache allow additional servers to be added to the cluster without having to reorganization or partition the data. The performance is such that there is linear scalability as each new cluster member is added to a maximum of 32 systems.

Software was then discounted for customers that chose to use the Parallel Sysplex as a clustered model over a multi system model that had non shared data.

New workload pricing

This was discussed earlier. The net is that specialty engines were defined that allowed no software license charges or deeply discounted license charges for Java, Linux and other workloads.

Hybrid Computing pricing

With the zEnterprise servers, the mainframe added the ability to have direct attachment to an IBM Bladecenter and in the process, have a dedicated communications channel and a dedicated management channel for these two devices. I already described how performance can improve when channel attached when discussing the SAP R/3 deployment between AIX and z/OS.

The IBM Data Application Accelerator (IDAA) is a query server, running on x86 blades that can get direct access to DB2 on z/OS. By shipping queries over to the IDAA server, read/write operations can continue on z/OS while read only performance of the query can be parallelized across multiple x86 engines and see performance improvements that could be 300 to 1000 times better than if run on z/OS. More importantly, the IDAA is “just an engine”. Security of the database and audit remains within DB2 for z/OS. Data copies over the network can be avoided. Some copying is inevitable, but flash copies can be made in moments, instead of long running data extract, transform and loads (ETL) to other platforms. This provides a significant advantage to completing near real time analytics and enables new decisions to be made, prior to completion of a transaction. There are too many cost savings to mention, but some are: less disk space, faster response time, improved security, better scale, less network bandwidth consumed, etc.

Mobile pricing

In the 1960’s and continuing through the early 1980’s, it was agents of a business that executed transactions. Travel agents, Bank Tellers, Claims adminstrators, ATMs and Point of Sale terminals. The consumer watched. These transactions were typically short, easily measured and predictable. The majority of the transactions occurred during business hours, such as 9AM to 5PM. Hardware and software were priced for the size of the machine dictated the overall software license charges.

The “Internet of things” has changed that. Consumers can now do things on their own, where an agent was previously required. They can start a claim, transfer money, deposit a check, buy tickets to performances and for travel. They can do this at any time of the day or night. They can query prices as often as they wish, waiting for the “sale price” to be good enough to actually purchase something. This is dramatically grown the number of transactions being executed and kept the Systems of Engagement and Systems of Record up around the clock. Any down time means potential and real loss of business. In April of 2014, IBM introduced the Mobile price usage to provide a discount for these type of consumer issued transactions. The goal is to make the monthly pricing more predictable and comparable with “commodity” platforms.

Summary

How bad it was (1990’s)

Let’s review how bad the mainframe was back in the early 1990’s.

  1. Water cooled, very large, heavy servers with large amounts of electricity and cooling required.
  2. An Internet network that was horrendously slow at the hardware and software levels.
  3. A Green screen Command oriented interface (similar to the DOS Shell that lived a very short while on PCs). Yes, you could “screen scrape” to make it look more graphic, but many considered that “lipstick on a pig”.
  4. A communication architecture that was inherently inefficient as it copied data between system components, many times, before it went on “the wire”. In PC LAN terms, it was a Ring 4 implementation (and worse)  instead of Ring 0.
  5. The wrong character set was used: EBCDIC, which required software changes and data conversions to and from ASCII or Unicode.
  6. Expensive software licenses for production and development.
  7. Proprietary, green screen oriented, development tools
  8. Proprietary programming interfaces
  9. Lack of Commercial Off the Shelf (COTS) new workloads

How much better it is today

  1. Comparably sized servers to commodity systems that use less electricity and cooling.
  2. An incredibly faster Internet connection for inter-system, intra-system and intra-cluster communications.
  3. While the command line interface is still available, a web service oriented management interface is now available.
  4. A communication interface that passes pointers to data and shares the data rather than copying it within a system image, across virtual system images and across a cluster of systems.
  5. Adoption of Unicode, ASCII and EBCDIC as base characters simplifies data consolidation on z.
  6. Technology dividends and a wide variety of pricing options make the TCO and TCA of System z competitive with commodity servers
  7. Single, cross platform tooling from Rational that includes mainframes, UNIX/Linux, Windows and Mobile systems (via the Worklight acquisition)
  8. Portability of applications within z/OS and Linux in a variety of open languages
  9. A far broader range of COTS software is available for Linux on z and z/OS

Some additional items that are better than commodity servers

  1. Shared data access, with integrity, scale and resilience,  across systems
  2. Shared Analytics and Transaction Processing to a single database (that can be shared across a cluster), while maintaining Service Level Agreements
  3. Hacker resistant (not Hacker proof) architecture that inhibits data and buffer overlays. The System Integrity guarantee has been in place since 1973.
  4. Capacity Backup licensing and acquisition dramatically reduce Disaster Recovery costs and procedures.
  5. System z hardware avoids 80% of the errors that might occur in a PC server environment. No down time nor failover would occur. System z software and hardware work together to drive System availability to 99.999%.
  6. Workload management of thousands of applications and hundreds of thousands of client connections enables dramatic cost savings over alternative servers.
  7. Incremental additions of software workloads without the need to install new hardware due to on board “spares” available to be turned on, on demand.

Summary

If you haven’t considered a mainframe in the last 20 years, it’s quite understandable. But if you don’t start reconsidering it today, you are making a fundamental mistake.

The modern mainframe is greatly simplified from where I began 40 years ago. I’m happy to say I  may have had a little bit to do with that 😉  Happy programming.