Multifactor authentication (MFA) and data encryption are the best means to reduce the risk of hacking attempts on a platform. However, it’s critically important to understand that an MFA solution targeted to one platform, say a mobile device or desktop as the system of engagement, is probably insufficient to address the security needs of another platform that gets accessed later, such as a system of record.
IBM’s acquisition of Red Hat has some interesting ramifications for System z. My last four articles on this blog were about Porting an Enterprise App to the mainframe. I provided details on the downside of using the mainframe for Linux. In particular, there are few binaries available and the user has to build their own code instead of acquiring the binaries from others. The main reason for this problem is that IBM has been afraid of being a distributor of open source since they first announced support for Linux in May 2000. This has been especially true for the mainframe. Heaven forbid that someone would make an IP or patent claim against open source code that came from IBM, regardless of the open source license that was distributed.
And even with that difficulty, the open source movement on the mainframe is succeeding very nicely. Rocket Software, where I am now working, has the Ported Tools for z/OS. They provide open source binaries because IBM wouldn’t do that. Better yet, they’ve been keeping those binaries up to date with the open source industry. At one point, there was a Redbook for Open Source on z/OS that included binaries, but those sample pieces of code were removed. This is what motivated Rocket to become the distributor. IBM has also used other sites, such as Marist College, to host any binary versions of code, in order to put a buffer between them and potential IP issues.
Red Hat is the largest supplier of open source code in the world. Most important to me, they include binary distros for the mainframe. This will mean that IBM will become a distributor of open source binaries for the mainframe. Now, if they only take that attitude and apply it to the other areas that they support. For example, the Linux Community team maintains a Github library for open source on the mainframe. Today, it’s mainly about source code changes and leaving it up to the user to build the binary, which can take hours, instead of the seconds needed on x86 platforms to download a binary. Wouldn’t it be great if this acquisition of Redhat led to IBM’s ability to create and post binaries for open source for the mainframe? The unintended consequence of this acquisition could actually make the mainframe even easier to use and deploy. That would be a fantastic next chapter in the amazing mainframe’s legacy!
At the end of 2016 and lasting a few months into 2017, I completed a proof of concept port of a large Enterprise Application that had been running on the Amazon Web Service Cloud to Linux for System z. This was a Docker based application written in Java…so of course, it would be trivial to port. WRONG. While the application is in Java, it called many pieces of open source code. Much of that code hadn’t been ported to System z yet or wasn’t widely adopted. What I thought was a very simple exercise turned into a six month effort.
What I’d like to do, via a series of blog entries, is share my experience in the hope this might help some other organization decide to do a similar porting task. While I’ve been working with mainframes for decades, this was my first Linux porting experience. So I’ll be describing how this experience helped me to Master the Mainframe, though that title seems reserved for university students.
This could be a book, but by breaking it up, it might be easier to understand.
The Basics: High level overview of the application, the development environment, the system set up required to begin the porting exercise and the scope of the port.
The Good: The people who assisted and taught me, the things that ported easily and the simplicity of getting started via the Linux Community Developers system.
The Bad: the new open source for System z, the modifications necessary to open source to run on z, the debug experience and the time necessary to complete the porting process.
The Future and Value. Regardless of the bad experience, there is a great business value in getting these types of Enterprise Apps on System z.
This entry is more about the basic desktop development environment and targeted production on x86 based cloud servers. This is the traditional development environment and primary target of the applications. I needed to fit in and work with this environment before I could ever consider doing the unique activities necessary for success on Linux for System z.
Because of the proprietary nature of the application and intellectual property, I’m not going to name the vendor or application. This overview of the workflow is simplistic, at best, so as to not give away any trade secrets. The vendor is an early start up with an application to handle biometric authentication in a marvelous way. This application has a callable interface to start a request and then, using cloud based services, does some communication with the end-user, does some analytics based on a number of system defined characteristics, logs a number of things for diagnostics, audits and future analytics, provides a go/no-go decision back to the original caller and has a number of applications and user interface applications to manage the cloud deployment. Finally, they have an enormous test suite to emulate and automate the entire end to end workflow.
This vendor was doing all of their development for the x86 platform and originally with any Linux version supported by Amazon Web Services. This included Centos/Red Hat versions. Their first development environment used Maven tooling and pom.xml scripts that targeted deployment into Docker containers. They used Github capabilities to clone and manage the source code libraries within their business.
The first major effort was for me to establish a development environment on my computer and prove that I could work with and build a workable x86 version of the code. My computer of choice was a MacBook Pro 2010 model running the latest MacOS at the time. First thing to do was turn my MacOS into a real developers machine. I installed xcode, Atom, SourceTree, Filezilla and Docker which enabled me to look like a Linux system, edit source files intelligently, manage access to the source files, facilitate cloning of source and execute the code. There were other local variant software that I needed to install using a script that was provided to me. I love the Mac, as did the vendor, who’s entire team used it, so that was really helpful. I then needed a VPN into their system and I was off and running. I used this set up for about two months. One thing I learned, painfully, that the 2010 Mac was SLOOOOOWWWW. What would take 15 minutes to do for them might take me over an hour. So I decided to upgrade to the MacBook Pro Touch Bar quad-core 16GB memory laptop. Now my work completed faster than their 15 minutes, which was a blessing. I can’t stress enough the value of a good starting point on the desktop or laptop for this type of development! It was life changing to me.
Open Source and Operating System Dependencies
The first version of the vendor code used Centos/Red Hat as the target deployment environment. This code runs over 50 Docker containers. Each container is intended to be as small, memory wise, as possible, so it is scalable in a largely virtualized environment. As mentioned earlier, they also used Maven and pom.xml scripts to do their container builds. Each container had a script that would gather necessary pre-requisite open source parts, their Java code and then do the build so there was an executable container. Naming conventions, versioning and more were part of these Maven scripts. 90% of the open source code used was available in a binary form as either an RPG, ZIP or TAR file. Those binaries were either copied into the vendor’s library system or accessed via a URL and dynamically downloaded from the internet during the build process. I’ll get into the System z ramifications of this in the Good and Bad blog entries.
This is the development environment I began my first phase of the port. The prototype I was building was only for a functional test to prove the code could work. We intended to accomplish our test goal with only 40 of the 50 containers being ported. We completed what we thought was a good test level of code after a few weeks of my porting. But then we identified some critical test containers were missing. Unfortunately, the vendor didn’t use the same library management rigor for their test suite and I was going to have to re-base my code.
Rebasing the code and changing Development environments
Unfortunately, that was the tip of the iceberg in changes. I mentioned this was a startup vendor. They had two very large customers that were testing the code when I started. They realized they had a scaling problem, early on. They also realized they had some development inefficiencies. When you get a RedHat, SUSE or Ubuntu distribution, there is a lot of software in the package, like getting the z/OS operating system, MacOS or Windows. As such, the kernel of the large distribution Linux systems can start at 250MB and easily be over 750 MB’s. When you add 100’s of virtualized containers, each having that size as the basic footprint, the overall system runs out of memory pretty quickly. However, if the kernel can start at 18MB and run about 50MB, then greater scale is possible. As development of this application began, the Alpine Linux distribution began and it met the small size requirement. The vendor began to rebase all of their test code and as much of the open source code as they could on Alpine to take advantage of this reduced memory benefit. That was and is an excellent business decision on their part.
Maven is a fairly complex environment for building docker containers. It works. Both the vendor and I proved that it could work. However, in addition to open source code, there are now open docker containers that can be leveraged, as is, to be included in place of an open source binary. However, in order to do that with Maven, the Docker definition files of these open containers must be cut and paste and then modified as part of the Maven script syntax. And each time the container definition changes in the open source world, the Maven scripts need to be hand modified. So the vendor dropped Maven as the base for their container build environment and switched to using Docker build definitions directly. Again, I applaud the vendor for doing this. It simplified the development environment, it gave them access to additional open source code repositories and made everything easier to manage.
The unintended consequences of the vendor’s change from Maven to pure Docker and Centos/RedHat to Alpine was I had to start all over on the port. I’m going to save the details of that for the Good and Bad statements as they are directly applicable to System z.
As far as Linux for x86 cloud environments, this vendor has a world-class development environment, working to create the most reliable, secure and efficient application possible. Ultimately, those attributes must apply to System z deployment as well. I’ll be covering that status in the other blog entires.
I provided a simplistic overview of what I intended to port to Linux for System z in Part 1. The original application was built for x86 systems. As such, all binaries are built to run on x86 systems. The Docker containers that these applications run in are x86 binaries as well. So my job was to create the Linux for System z (aka S390X) binaries, with as little change as possible.
I also mentioned that this was a start up vendor with whom I was working. I had done some business work to show them the value of porting the application to System z, but they were neither skilled in, nor able to afford their own System z. So I gave them the challenge to let me prove to them this could be successful and they took me up on it and agreed to work with me.
Vendor Development Team
While a small development organization, they still had over 25 very proficient programmers and testers. I was extremely fortunate to have their lead developer as my mentor. He and I would meet at the same time, for an hour every day to check on progress, educate me or diagnose any problems I might have so that I could make progress for the next day. Most important is he was learning about the mainframe and intrigued by the possibility of business success as I was, so it was a great experience for both of us. I greatly appreciate the time and effort he put in to make this a success.
Linux Community Development System for z
Where do you find a mainframe? You ask the Community Development team. Eva Yan at IBM was instrumental in approving the vendor and I to get access to Docker containers on the mainframe. Cindy Lee at IBM was fantastic, with her team, to help show me where all the open source for z was available in the community and Martha McConaghy at Marist College, the host for vendor access to the LCDS was terrific in helping me to keep the system running.
Docker is a great place to work with portable code. My development environment was an x86 Docker container environment that pointed to the S390X Docker on the LCDS system as the target deployment environment. I’m not going to spend time giving you the details on the set up, but suffice to say it all works well.
I didn’t mention before that the vendor is on a different continent. So imagine from my laptop, a VPN to the vendor’s libraries where some code is downloaded, merged with code on my desktop, Docker on my desktop puts all the parts together, ships it securely to the Docker on the mainframe image, does the build and sends results back to me. So if this process took 10-15 minutes to do on my laptop, suffice to say, when you add up the networks and bulk distribution of code between systems and do the build, it’s going to take more time than a single system. Doing a single container build, for the first time, was never correct. My mantra, for years, has been “Next time for sure!”. I’d fix what needed fixing, get a little farther the next time, repeat the mantra and try again, until finally, I’d get a successful build. The time or performance isn’t a problem when building a single container. It’s when you build 40-50 containers at once, or as I liked to call it “The Big Bang”. Then it was hours to do the build on the mainframe, instead of an hour on x86. You’d think that was the bad, right? It was good, because a call to Eva, requesting some more memory and processors and I moved to a very competitive deployment environment. For just like my MacBook 2010, which was under configured for this scale of development, the initial Linux system I was given was an under configured virtual machine. With a simple config change, within moments of my request, and literally no down time, I was up on a larger Linux image, due to the magic and wonders of the underlying scalable z/VM server image.
Open Source Access
The LCDS virtual images came with RedHat kernel as the base, with some optional software included, but that was all. I need several dozen pieces of open source software to add to my environment to build my S390X binaries. Again, I don’t want to spend the money to buy a supported Linux distro for this Proof of Concept. I’m directed to Sine Nomine Associates, and in particular to Neale Ferguson. He could not have been a better ally in this effort. First and foremost, he pointed to libraries on their servers where I could retrieve many of the binaries that were necessary. It was such a relief to find many of the rpm’s I needed on their website. As mentioned earlier, I was a newbie to this kind of porting. He spent considerable time mentoring me on both basic Linux and System z specifics to keep me moving along. As important, Neale was on the Docker band wagon. He’d begun building docker containers with specific functionality. I was able to take several of his containers and imbed them into the containers I was building to simplify my deployment.
The Linux Community also has Github repositories of System z ready open source code. I bookmarked those pages and visited them often. I’m pointing links in a Bibliography in Part 4.
The real dilemma came when the vendor switched from Centos to Alpine as the base Linux kernel. Alpine was so new in late 2016, early 2017. While both are Linux derivatives, the syntax of packaging applications is different. As such, Docker builds for Centos are different from Alpine. Because I was doing a proof of concept, it really didn’t matter whether I used Centos or Alpine. However, the longer my porting took, the faster the vendor was converting their code to Alpine, so now, I would have to make “throw away” changes to support Centos.
Worse than that, there was only one person even trying Alpine on the mainframe and that was “some college kid” as a research project. How could I build an enterprise application on a system that one unpaid person was supporting? That person was Tuan Hoang and I am indebted to him. He was a Marist College student. I began contacting him late in 2016. While he had the kernel ported, there were very few packages for Alpine ported to S390X. He was quickly up to the task. I gave him a list of high priority packages. Each night, I’d get an update of what he completed. Each day, I’d build some more containers off his evening’s work. It got to the point that only third-party open source packages were not done by him. This really got my development effort going. But the best news of all was at the end of my project. Tuan had worked so hard to get his “prototype” of Alpine for System z going that the Alpine community accepted S390X as a primary target platform. All Alpine packages would be available on S390X, simultaneously to their deployment on other hardware architectures. It was painful, but it was wonderful at the same time.
Good people make life easier
What I found throughout this porting effort is there is a wonderful community of people dedicated to the support and value of System z. They were very accommodating and helped reduce my efforts greatly.
As I’ve explained in Part 1 Basics and Part 2 Good, I did a proof of concept port of an Enterprise Application from Amazon Web Services on x86 to Linux on System z in 2017. The good news was I got to the point I needed to, the bad news was it was more than difficult to get there.
Linux is not Linux
Open Source is open source…available to anyone. The story goes that Linux is Linux. Close, but not quite. Unfortunately, architectural chip bits (Big Endian vs Little Endian) is one of many differences and there is code that needs to change to handle these differences. There are also supported platforms, “tolerated platforms” and unsupported platforms. This is the problem with Linux on System z. The marketing hype is that all of Linux is supported on z. The reality is somewhat different. Not necessarily insurmountable, but you better know what you are getting into.
When Linux on z is a supported platform, then the packages for System z are supported in binary format, such as an RPM file for Centos/RedHat or an APK file for Alpine. This is the best case and makes development of S390X on par with other platforms like x86 and ARM.
In this case, the code may work on S390X, but it’s a source code build. You can find instructions on Github for S390X as to how to modify the code to get it to work on the platform. But if you want to use that code, it could take a long time to
Do all the things necessary to manually modify the code
execute the code to create a binary.
BTW, when I complained to IBM leadership about the lack of support for Couchbase, they suggested I use a different, easier product that was available on z. Since I was porting and not a true developer, this was not a possibility for me. I had begun negotiations with Couchbase toward this goal, but stopped working on it when the prototype ended.
There were two cases where neither the open source community nor the Linux on z community had guidance on how to get a particular open source program on the mainframe. In those two cases, I was able to get through the code, successfully and get a binary for System z. The good news was it was pretty simple to do. I was quite fortunate. If it hadn’t been easy, this could have ended the project earlier than I had hoped.
Docker containers are not portable across hardware architectures
I’ve seen some hype that once you get it in Docker, it’s portable to any Docker. I’ve heard a few mainframe customers believe any Docker container can run on System z. I’ve also seen articles in IBM sponsored magazines that purport this to be true. This is a combination of marketing hype and misunderstanding. It all depends on the container architecture/binary and source code. Typically, a container binary for a particular architecture, such as x86, should run in a Docker container on any x86 platform, even if it’s a different operating system running Docker. For example, Docker running on x86 version of RedHat 7.3 could be running containers with RedHat, SUSE, Alpine, Ubuntu, etc, as long as they were built for x86. Similarly, I ran Docker on a RedHat 7.3 image for Linux on System z, and had containers with Centos and Alpine running with binaries for S390X.
The only containers with source code that were portable were built exclusively with interpretive languages, such as Java or Python. Those could be portable across hardware architectures. Many of the test cases used by this vendor fit into that category. However, as soon as one of those interpretative languages makes a call to open source code middleware (e.g. Couchbase), then the container is no longer portable across architectures because the middleware is not supported across architectures.
When I started this project, Docker on z was pretty new. Once in a while, it would have issues. Only a couple of times did it require Marist College to restart my z/VM guest. The other times, it would automatically recycle itself and get running again. I believe it’s improved since we began the port effort until now, but it’s been a few months since I tried it. I’ve heard from others, though, that the experience is better. During our Big Bang builds, we would peg each of our System z processors at 100% busy for a few hours. The fact that it would stay up and continue processing is a testament to the reliability of those large code tests.
Ultimately, I have a wishlist for the Open Source Community on z:
Where source code changes are necessary, such as with Couchbase described earlier, supply a Docker build file to automate it for anyone that wants to do the build. It would be so much faster.
Continue to lobby third-party open source middleware providers to support system z. In many cases, it takes a vendor, such as I was working with, to create that business case jointly to get it done, but doing that will lead to more usage on the platform. If you build it, they will come.
Create more binary packages instead of source code update files. It greatly reduces the development time necessary for z unique porting. The more extra work necessary to support z, the less likely the x86 people will move there.
The net of all this bad is the initial effort to support the mainframe is longer than it would be on x86. However, if you have the patience to get to Part 4: The Future and Value, you’ll find that you should be rewarded for the effort.
In Part 1 The Basics, Part 2 The Good and Part 3 The Bad, I’ve explained I did a proof of concept port of an Enterprise Application from Amazon Web Services on x86 to Linux on System z in 2017. The good news was I got to the point I needed to, the bad news was it was more than difficult to get there. But why did I go there in the first place?
The vendor for the Enterprise application was targeting the Financial Services industry for their initial deployments. This is the primary customer for IBM System z. Their beta customer is running z/OS transaction processing via CICS, but wants to authenticate customers using this vendor’s product running on Amazon Web Services. In order for CICS to call the AWS Cloud, it has to launch Websphere on z/OS to call the vendor’s service on AWS. The vendor’s application has to do it’s task of authenticating users and get all the way back to CICS in less than 18 seconds so the transaction doesn’t time out. It’s a really powerful use of the vendor’s application and valuable to both the consumer and financial institution to avoid potential fraud or cybersecurity scams.
Java and Analytics run better on z/OS
I was told this vendor wrote all their code in Java, so I immediately began a plan to get this running within z/OS, since Java runs so well there, especially on the z14 systems. I also knew that in the time allotted to run on AWS for those 18 seconds, only three biometric/analytic tests could be completed on behalf of the consumer. I hypothesized that if the vendor app ran within z/OS perhaps up to ten analytic tests could be completed using the outstanding analytics and Java performance. However, once I learned of the number of open source middleware programs required and the complexity of porting them to z/OS, I went to Linux on System z as the target port.
Linux on z as a private cloud has more value than a public cloud
Using RDMA as the memory based communication between z/OS and Linux LPARs, I know it will take a bit more time than running inside z/OS, but much less time than going to a public cloud, so I hypothesized that eight analytics tests could be done instead of the three on AWS. And regardless of z/OS or Linux on z implementation, the vendor agreed that the software price would be the same as AWS. The net is, z would have additional analytic value, and given it’s hardware and software integrity and reliability, it would offer better security and business resilience than any public cloud provider.
So that’s what I set out to prove. Sadly, I got so close and the vendor changed their mind on their business strategy. They received a significant new round of venture capital investment, signed up several new financial firms to try their code and they decided to stick to their current cloud plan and stay off the mainframe, for now.
I still believe that my hypotheses as to the performance and value were correct. But the activity ended just before I was able to prove that. However, the exercise did confirm the possibility of getting the product on the mainframe successfully.
Docker inside z/OS? That would simplify things!
But what else is possible? I said in Part 3 that Docker containers are not portable across architectures. However, they are portable within the same architecture. There are some prototypes underway for Docker to run within z/OS. Given the way Docker works on other platforms, it would infer than any Linux on z containers could run unmodified within z/OS. If Docker for z/OS were to run on a zIIP processor, there would be no software license hits for z/OS. If that all comes to pass, that could lead to significant transaction and analytic value within z/OS and greatly simplify the system management requirements for these types of hybrid workloads, while improving the overall security, resilience and performance and reducing the operational costs. I would hope that a public announcement of this capability is not too far in the future.
Savings and Operational Strengths
That, my IT friends is a win for everyone. Any of the bad associated with a slightly more complex development environment can quickly be eradicated with a greatly reduced operational expense that has greater operational benefits than any alternative architectures might try to demonstrate. This type of workload makes for a very compelling end to end benchmark comparison as well. So while I didn’t succeed in getting the enterprise application to market, that was because of a business decision rather than a technological impediment. And the business decision was tactical, based on their new financials.
I learned a lot and documented many of the short cuts I took and set up required to make this development effort possible. I’m happy to share the experience if you’d like to undertake your own development effort. While I thought the end of the project was a failure, it’s unintended consequence, with the efforts of the great Linux for z community identified in Part 2, is that this will be easier porting for everyone that follows.
What’s a bottleneck? From Dictionary.com, it’s “a narrowentrance,spotwheretrafficbecomescongested”. In IT terms, it’s something causing slower operations or that inhibits a Service Level Agreement (SLA) from being met. The worst case scenario is a lot of IT shops are absolutely confident that they don’t have bottlenecks as they are meeting or exceeding their SLA’s. They couldn’t be more wrong!!!
Yet, none of these will find the modern IT system bottleneck. When you have an IT system bottleneck, there’s always someone to blame. But who is it? Is it the System Programmer’s fault? Is it the Application Developer’s fault? Is it the asphalt? Oops, wrong punchline. No, it’s the System Architecture’s fault. It’s a 1990’s mentality that looks at IT in operational silo’s and independently manages the systems. But hang in there for another moment. There is a cure.
The 1990’s methodology bases IT operations on server silos. The mainframe is independently managed from the Unix servers, which are independent of x86 servers, which are separate from cloud and mobile and desktop and network. Security is done for each domain. Business resilience is done for each domain. Budget’s are created and departments compete for more spend in their particular area. Some areas might claim they have a bottleneck and warrant more spending to resolve it. Next budget cycle, they’ll still have issues and want more.
Another type of silo-ed operation is looking at separate systems for Record, Insight and Engagement. Systems of Record are the master database and transactional systems that update those databases (e.g. credit/debit, stock sales, claims, inventory, payments, etc). Systems of Insight are the analytic systems (e.g fraud detection, sales opportunity, continuous flow delivery, tracking). Systems of Engagement are the human computer or Internet of Things (IoT) interfaces (e.g. mobile, IoT device, tablet, browser). Many businesses create silos to manage each of these areas independently because if you had ever tried to do this in the 1990’s, you’d hit a bottleneck or drive up IT costs too high. Funny how the systems of the 1990’s actually created the hidden bottleneck today! But it can be fixed.
Where can you buy the “fix” for this? Is it via a software product? No. Hardware product? No. Cloud? No. Consulting services? Maybe. But the reality is every business can solve this pretty easily within their own environment. I guarantee that your business can far exceed current SLA’s and establish new business goals. In the process, your business can save tremendously in IT expense, while improving security and business resilience. The solution is pretty simple.
Stop copying data between systems! In the new API economy, all of the systems have been modified to allow for direct access to applications and data from other systems. The change is either philosophical and/or organizational for most enterprises. It’s all about managing the IT systems together instead of separate silos. That starts at an architectural level, with hybrid development systems and extends to hybrid operational systems that address end to end security, business resilience and performance.
If you’ve moved data to another server to keep the Systems of Record separate from the Systems of Insight. Stop the move. Keep the data together. Systems like IBM’s mainframe are now capable of hosting both databases and analytics in a single system and improving analytic performance many times over separate Systems of Insight without impacting the SLA’s of the transactional systems. The applications that access the Systems of Insight can be easily modified to point to the Systems of Record instead via updated device drivers without changing any code logic. This changes things like batch analytics, which might be used for fraud detection into real time analytics that can be used for fraud prevention. And in the process, businesses will save with reduction in storage, network bandwidth and system utilization, costs and time associated with copying the data. Products such as Rocket’s Data Virtualization Studio can provide the device drivers and mappings necessary for applications to share data from a variety of Systems of Record, across platforms. And new apps can be developed to join the data from different sources, including partner organizations or from “the cloud” to solve business problems in new and creative ways. These applications wouldn’t be possible without sharing data. Apache Spark technology is one means for collaboration across data sources.
There is no reason to copy data to move it closer to or tailor it for a specific System of Engagement. The API economy allows for applications to directly access the data or transactions on other systems via the API economy. New pricing options are available that allow for increased transaction rates, due to direct access to mobile, at a lower cost than traditional access methods. zOS Connect is one of the tools for making the API connection between mobile and transactional systems.
Regardless of how you might transform your business, the unintended consequence of standing still on current IT silo-ed operations is there are bottlenecks and slow downs in business systems that depend on heavily copying data and batch windows to facilitate copying. Direct access to data and devices is the future. The future is now. Begin the migration to hybrid operations management. If you need help in deciding how to look at your architecture differently, don’t hesitate to ask me.
In an earlier post, I wrote about some of the things to know before and during a trip to Cuba. It was a unique experience for my three travel companions and I. Everyone has a different experience, depending on their expectations going in. We expected a bit more “touristy” stuff, given that Europeans and Canadians have been going to Cuba for many years. But we also expected it would not be like any other experience and were not disappointed. The following is a summary of my experience there. The ratings of the experiences are my own and may not be shared by my travel companions.
The Very Good
I’d classify this category as I’d be happy to do it again and at any time. Each of these offers a good reason to return.
Meeting The People of Cuba
In general, we found all the people that we had conversations with to be extremely pleasant and courteous to us. Hotel employees, restaurant servers, people we asked directions of, including police, farmers and small town residents. This is a very poor country. Education is free and good, going right through to graduate and medical school. There are many talented people in the country. But those jobs don’t pay well. An engineer or optometrist might make 30 CUC a month. (1 CUC = 1 us $). A doctor might make 40 CUC a month….a month….It’s not a mistake. And they might only work 2 days a week at those jobs. So they take tourist related jobs where they can get additional income and tips. We met an optometrist that cleans apartments. Engineers and doctors that drive taxi’s. It was heart breaking to see this and hear their stories. They were genuinely nice people. Food there is expensive. Many live day to day on rice and beans. They work in hotels and restaurants so they can get better meals and maybe even bring food home.
There were many beggars in Havana. There were also people that would be extremely nice, only on the hope that they could receive a tip from you or a commission from the person that they brought new business too. However, the good far outweighed the bad here. We generally felt very comfortable, regardless of where we were traveling.
I mistakenly left the B&B with one of the two sets of keys. There were three keys to the apartment. I was in Varadero when I realized the error. I texted our host and she told me to leave the keys are the rental car stand at the airport. Honestly, I’d have a hard time doing this in the US. My prejudice would say to never do that in Cuba. At the counter, which is more like a glorified lemonade stand with a main person and several others hanging out to move cars, the counter person took the keys, put a label on them, stapled them to a rental car brochure. I wrote the host name and phone number down. He told me his name and said to pass it on to her. Four hours later, she stopped at the airport and got her keys. A wonderful example of the kindness of the people of Cuba.
Airbnb – Casa Colon in Vieja Habana
There are three parts of this review. 1. Apartment. 2. Location and 3. Host. The Apartment was wonderful. Two queen size beds, a lovely living room, patio and kitchen. It was very clean, well stocked with snacks and drinks with a mini-bar hotel quality that made us very comfortable to be there. It was located on the fringe of Vieja Habana (Old Havana) and allowed us to walk throughout this area. We could also easily find cabs to and from the apartment. Cab drivers can easily find the apartment as it is next to the Parkview Hotel, which we used as our reference point.
Our host, Ana Travieso, and her staff of cooks and cleaners were phenomenal. Ana picked us up at the airport on our arrival and then took us back to the airport and our next hotels on this trip. The breakfasts were very large and wonderfully prepared. There was something, or better said, a lot of things available for any type of breakfast that someone can imagine. Ana also provided us with local tourist and restaurant recommendations that were extremely valuable and spot-on to what we were looking for on this trip. Havana is a great experience but not a paradise. Ana provided us with the knowledge to make it an excellent experience. Note: Leave your computers behind when traveling to Cuba. Wifi is a paid utility via cards that you pre-purchase. Price varies from $1.50 -$3 per hour. That’s everywhere and not unique to Casa Colon. We were able to use the wifi here when we wanted.
Living Room at Casa Colon
Bedroom at Casa Colon
ChaChaCha’ Restaurant in Vieja Habana
At Ana’s recommendation, our first meal, a late lunch, was here, just two blocks from the apartment. It was tremendous. The Shrimp in Garlic was delicious. Another winner was the Lobster salad. A huge and delicious portion. This was our second favorite restaurant.
Lobster Salad at ChaChaCha’
Dessert at ChaChaCha’: Chocolate sausage. Go figure.
El Del Frente Restaurant in Vieja Habana
We were trying to get to the restaurant known as 304 O’Reilly or O’Reilly 304. It’s #304 on Calle O’Reilly. As American’s this looks to be a good Irish name. However, to locals, it is pronounced closer to Orelia. Thank goodness for the hard copy map. Well, we couldn’t get in. But the server suggested the restaurant across the street at #303 and owned by the same company. The server there said if we were willing to eat fast and be done before their reservation in 50 minutes, we could stay. He agreed to serve us quickly and then did just that. It was a phenomenal dinner. There was a watermelon mojito that was fantastic and ordered by two of us. The others got Pina Colada’s. In a second round the Colada drinkers ordered the mojito’s. They came without Rum. When we notified the server, he came with the bottle and gave it to us. Three of us got three different varieties of delicious Taco’s. The menu only says Taco’s. However, when asked, there were Lobster, mixed seafood and meat Taco’s. We got one of each and shared. All great.
Mojito’s – the drink
When in Cuba, the Mojito is king. We had them everywhere of the basic variety, with one exception for a Watermelon flavored one. All were good. We generally had no problem ordering four mojito’s at a time, other than breakfast. They take a bit more time to prepare, but it’s worth it.
Mojito. National drink of Cuba
Watermelon mojito at El Del Fuente restaurant.
Bridge at Mirador de Bacunayagua
This is a rest area overlooking a bridge that spans a very large valley. It is on Via Blanca, the main road between Havana and Matanza’s along the shore, en-route to Varadero. The real attraction was the Pina Colada’s they serve at the stop. They give you the bottle of rum to mix to your own consistency….Excellent! We acquired some souvenirs here as well. Parking was 1 CUC, but worth it.
At the scenic vista, there were a number of large birds, probably vultures, riding the air waves. They were above and below us. It was a spectacular vista.
Beach in Varadero
I’m no beach expert, but this was the nicest beach I’ve ever experienced. I was told that this is true of the North shore of Varadero, so other hotels would be like this too. And I’m not talking about bars and sailboats and the extras a hotel might offer. This is pure sand: no rocks, no shells. It’s got a gentle slope into the water where you can walk out quite a bit. There was some surf for occasional body surfing, but nothing scary. There was a wind that blew this fine sand along the beach. We were told that’s abnormal for the area. You’d better cover any food and drink to avoid chewing sand. The air temp was 85 F. The sun was out. But the sand wasn’t blistering hot. It was very comfortable. The downside? We brought a lot of that fine sand into our hotel room.
Panoramic view of Varadero beach at Barcelo Solymar Hotel
Palapas. Palm umbrellas at Varadero
Buying Rum to bring home
There was a Rum store in Varadero close to our hotel. It was a substantial discount to the rum and cigar store at the hotel. We got 5 half shots of rum to taste test. I never knew there was chocolate and coffee rum…well, I guess if you can do that with Vodka, you can do it with anything. Needless to say, we bought several bottles for the trip home. Don’t forget you’ll need extra time, and maybe money, to check the bag with the bottles and then wait 30 minutes at the carousel at your destination.
Our other experiences – the Good
I guess I’d have to say that these were good enough to do again, but not necessarily the reason to do it all again.
Museo de Bella Artes in Vieja Habana
Interesting museum across 3 levels. Very large spaces. Most of the art was post 1959. Many of the styles mimicked those of the US during the same period. There was also some “ancient” art from the 18th century on. Works similar to Gilbert Stuart, so matching that timeframe as well. A lot of pro-Castro and Che Guevara in this collection. It was a reasonable cost and interesting viewing. It was conveniently located one block from our Airbnb.
Museo de la Revolucion
This was more of a military museum. Missile launchers, tanks, helicopters, Jeeps (Willys) and other instruments of destruction outside. Inside a glass enclosed building was Fidel Castro’s yacht. Supposedly, for 8 CUC, you can walk into this park/museum to see it. If you walk around the park, you’ll see it all for free. Note: at night, there are soldiers on each of the four sides of the park to protect the museum. Interesting. After reading the weblink I added, I see there was a palace next door that we could have visited as well. That would have been more interesting than the military equipment.
Plaza Vieja for the Cuerdo Vive concert
It’s a nice old plaza, in a European mold. A nice restaurant, Factoria Plaza Vieja on one side. Street vendors in the corners. I love the Maiz – corn on the cob nicely seasoned and on a skewer to eat. Didn’t get one this trip, but I was drooling for one…The lines were too long. While there, they were setting up for a concert: Cuerda Vive 15th Anniversary. It translates to Cord lives, but featured acoustic guitars. That was on Thursday. On Friday afternoon, we saw them broadcasting on TV. Each performer got two songs. We arrived four hours later and the plaza was packed and the music continued.
Hotel Nacional – Malecon Havana
The treat here was to see the sunset on the Malecon and have a mojito from one of several outdoor bars on the property. This is the most famous hotel in Cuba. Celebrities and dignitaries stay here. It was very well maintained as it has never gone out of favor. The drinks were good, but pricier than elsewhere. We arrived 5 minutes late as it is not a trivial place to drive into. However, two drinks later and we were well prepared for dinner.
lawn at the Nationale on the Malecon at Sunset
Paladar Vistamar in Miramar Havana
This was highly recommended. A Paladar is typically a family owned restaurant in a home. This was a traditional restaurant with three floors. It was attached to what appeared to be a defunct hotel. One level surrounded the empty hotel pool. We ate on the top floor, outside. This is situated just above the north shore of Cuba. It was a beautiful evening and a decent meal. We had to make a reservation here. It was packed.
Barcelo Solymar hotel in Varadero
We originally booked our trip, via GalaHotels at Be Live Experience Las Morlas. In early February, we were informed, via a cryptic note, that we’d been moved to Barcelo Arenas Blancas which adjoined the Solymar. We decided to stop at Las Morlas to see why we’d moved. It seems someone had cancelled the reservation in December. I’m guessing GalaHotels scrambled to get us a new place in early February. We showed up at Arenas Blancas and were told we were booked next door. Well, we lucked out. The quality of the pool, bars and facilities of Solymar were superior to Arenas Blancas.
Like the rest of Cuba, this hotel was aged and needed some repairs. However, it did seem to be getting some attention, though not soon enough to cover all the blemishes it had. The lobby had vines growing that were five floors long and provided wall to wall covering. That must have required a lot of maintenance. However, the hallway lights going to our room were out for the entire corridor. I could imagine a single woman getting a bit nervous going through there themselves.
The room had two queen beds. Our shower needed three door panels to keep water out. It only had two of the three and the missing one was closest to the shower head, so unless you removed the head and stood behind the second panel, water went all over the floor. There were three pools between the two Barcelo hotels. The pool directly by Solymar was the best and the only one we used. It was nice and comfortable. The beach was a short hike from our room. The beach itself: sand and water, was one of the best tropical beaches we have ever been at. The sand is fine, there are no rocks or shells. There is a gentle slope in the water and you can walk out quite a way. While there, we had a decent surf and fairly strong wind. This was wonderful. The beach has a number of “palm umbrellas” known as palapas. These were pretty old and poorly maintained. They weren’t much of a sun blocker due to the number of missing palms. The beach bar was terrific and there were a number of recreational activities to take advantage of at the beach. The hotel is all inclusive. There is a large buffet restaurant, with plenty of seating, in each hotel and five separate restaurants that require reservations. Book reservations early, if you want to go to one. We went to the Seafood restaurant once. There were three main courses to choose from, a soup, salad buffet and dessert. Honestly, when we left, we all agreed that the buffet would probably have been better. The buffet covered three meals a day. There were many, many choices. The servers were terrific. The hot food stations, were food was cooked to order, was probably the best, as the food was hot. Other stations, where the food was in pans, was not so hot. And you’d recognize some of the food from the evening before. Fruit, breads and drinks were very good. All of the hotel employees provided fantastic service to us and answered any questions we had. We greatly appreciated their efforts.
Pina Colada’s – the drink
We figured this is a tropical island, so Pina Colada’s would be native. Cuban’s consider this an international drink. So they charge a little more. The important thing to note is how they are made. Several use powder mixes. Ugh. Some didn’t use creme de coco, so it was just Pineapple juice and rum. Not bad, but not too sweet. Others were fantastic. And a variety of places split an cored a pineapple, put the drink in the core, cut and notch in the top and provided the whole thing to you. In one case, they gave you the rum bottle to “season to taste”. That meant drinking the virgin Colada enough to add room for the rum. If all of them were as good as those, this would be in the Very Good category.
While I didn’t participate in this activity, those that I was with did quite effectively. No hidden Picasso’s but there are plenty of prints to acquire in Cuba. Just make sure they aren’t the factory models that are mass produced by forgers/copiers of the realm thing.
Again, I didn’t participate, but we got tremendous advice from some large cigar shops to some small ones. There were always people on the street trying to sell us “local varieties” that were rummage sale carryovers/forgeries of name brands. Ultimately, we acquired some premium brands, but they were not cheap. As much as 25CUC per cigar.
Walking along Paseo di Marti (aka Prado)
This is the main drag that separates Old Havana from central Havana. The Capitol is there, a number of museums, restaurants and other tourist attractions. It’s a divided boulevard.
Easy to walk, wide open and lots of picture opportunities.
In Cuba, automobiles have the right of way. If you plan on crossing a street, beware. The drivers seem out to get you. If the light is green ahead of you and you feel it is safe to cross a side street, you’d be wrong. A turning vehicle could hit you. This is a sharp contrast to NYC where the pedestrian has the right of way. It takes a bit to get used to.
Our other experiences – the not so Good
Not the sole reason for not returning to Cuba, but these are things that catch you attention in a negative way.
Through out the cities and country side are billboards claiming, in large print, that Castro will forever be with us. Long Live the Revolution. Che (Guevara) and Fidel.My guess is Fidel was beloved when the revolution occurred. But given the income and environment today, most people weren’t too pleased with the results.
One local told us there were 2 million people in Havana. 1 million residents and 1 million policemen. Everywhere we went, there were National, City, local and military personnel. One the highway, there was a motorcycle policeman about every 10 kilometers. It was both intimidating (we didn’t speed) and reassuring in large crowds. But it did give us pause as we saw so many of them.
Sloppy Joe’s Restaurant
This is a tourist trap that’s very well done and close to the National Capitol (Capitolio). It was also close to our apartment. The drinks were okay and the food was passable. But it was also more expensive. It wasn’t worth a second visit. But lots of celebrities have been there.
San Jose Market – Vieja Habana
This is a huge market with many cubicles, similar to some of the open air markets of NYC. However, you could classify the cubicles. Wooden toys. Clothing. Cuban memorabilia. And the contents of each were the same as a dozen other cubes. This merchandise was all mass produced. It was the same stuff that we saw in markets and private home stairwells throughout the city. None of it was worth it.
Walking along Calle Muralla to Plaza Vieja
One of our party got very nervous walking down the street. It’s a run down street. Many private homes selling wares and begging you to come inside. A mass of people going up and down the street with the majority being locals and not tourists. It’s easy to see how intimidating such a place can be, given concerns going into the trip. However, as stated earlier, it was a very safe area. Locals are punished heavily if they do anything negative to tourists. So the reality is, it’s a tolerance. But if large crowds of locals concern you, head down the parallel streets of Obispo (large market dedicated to tourists) and O’Reilly instead. More tourists than locals there. Maybe even less crowded.
Hotel Copacabana – Miramar Havana
This was a functional hotel. Kind of like Hotel 6. Not a lot of frills. It is located on the Malecon, with direct access to the shoreline. It was very different from what we expected. The hotel itself, was similar to much of what we found in Havana: time has passed and there were a lot of cosmetic updates that could be made. The rooms themselves were large enough and beds comfortable. The bathroom worked well. Overall, the interior of the building looked dirty and paint was peeling and doors were rusty. There are two pools. The fresh water, in-ground pool is large enough and comfortable while in it. However, it’s old, you need to be careful around the edges and the cement statues in the center of the pool have worn down, with age, to the point they are no longer recognizable.
The “natural pool” is a seaside salt water pool. It is aged and somewhat difficult to enter, due to a sharp drop off and slippery algae. You’ll need to use the attached rope to lower yourself into it. However, once in it, it was actually warmer than the fresh water pool. This really looked like a Roman ruin. However, it was still enjoyable to the people that took advantage of it.
The breakfast buffet was passable. Much of the food had been sitting awhile and was cool. This is something we found “typical” at other hotels we visited. I’d look for something else before staying there again. However, I would stay there again, if necessary, as at least I’d have a proper expectation for next time.
I was going to put this and car rentals under a category of bad – something you never want to do again. However, the reality is, if you ever want to travel in Cuba, you need to be able to navigate. Do not under estimate the value of detailed paper maps. Reality is, you need multiple maps of the same area. One with a high level view to show street navigation. And then multiple levels smaller that show the details of shops, restaurants and tourist destinations so that there is a frame of reference to easily find them. When driving a distance, you must come up with turn by turn directions that are better than the text of Google Maps. Even their mini maps don’t show the details necessary to navigate. So plot out some basic things you want to do. Once behind the wheel, you’ll need a good navigator or plan to stop often to check if you are traveling in the correct direction.
Can’t live with them, can’t live without them. As such, this can’t be bad, but it doesn’t have to be good either!
The problem here was the wait time to get the car and the quality of the transmission. Otherwise, we had a pretty big car, compared to what I thought we’d get. We got where we needed to get to, eventually. With the alternative being buses for long distance driving, the car rental was terrific. Know in advance that the wait can be terrible.
This is remarkably different than my usual IT related entries. Back in October 2016, I received an email announcing travel from NY to Havana will begin in December for $99 each way. I bought 4 tickets for February and informed my travel mates. Those dates weren’t good, but re-booking within 24 hours, without penalty, and we had the start of a trip.
While American’s haven’t been traveling easily into Cuba since 1959, Europeans and Canadians have been. As such, there is a tourism industry that we can leverage to find hotels and Bed and Breakfasts, as well as rental cars and tourism spots.
An interesting place to visit, but….
The net, from my travel companions and my visit was that Cuba is an experience. It is far from a tropical paradise. It’s a third world country that requires a tremendous amount of improvements and cleaning to come up to what we’ve become used to in the US. It’s something we are very glad that we did, but there will be no rush to go back. Maybe in 10 years after they’ve completed some of the planned updates. I can say that we felt pretty safe there. It was easy to get in and out of both countries but there were a lot of “lessons learned” from our experience.
Through friends, we found some great resources. One of the best was this blog: cuba-travel-guide-for-americans. I don’t want to repeat everything they said. Take a look at that for yourself. Here’s an update to some of that info:
We booked our round trip to Havana. However, we ended our trip in Varadero. In hindsight, we should have flown home from the Varadero airport, a 20 minute ride vs. the 2 hour trip we took back to Havana. There are many airline choices now, but not all go to the same cities. JetBlue, for example, doesn’t fly into Varadero. However, had I waited two weeks, I could have gotten the same price from American Airlines or Delta.
At JFK, there was a separate line, downstairs from the JetBlue normal counters exclusively for the Cuba flight check-ins. You buy the VISA there. It’s a pretty long line, so get there at least 45 minutes earlier than you would a normal international flight. There was no online check-in for either to or from Havana, because of this.
Customs into Cuba
Pretty simple. On the plane, you receive two forms to fill out. 1. About the trip and where you’ll stay in Cuba. One per family. 2. Medical facts about you. Each person fills this out. On the trip form, there is the Reason for Trip. We checked off Other, as Tourism is not yet a legal reason for entering Cuba. We were there in Support of the Cuban People. That is one of the 12 allowable reasons to enter the country.
One person at a time can see the customs agent. Do not go up as a family, though one person can take child(ren). They stamped the Visa, took half and stamped the passport. Your picture is taken. Receive the other half of the Visa. DON’T LOSE the second side of the visa that is stamped on entry to Cuba. You’ll need that to get out.
As you exit the terminal, you hand the medical form to someone in a white medical looking coat. They didn’t look at it. Just added it to a pile.
They then Xray your belongs upon entering the country. You are in.
Customs out of Cuba (Emigracion)
They take you picture leaving. Again, one person at a time to the Customs agent….why say this? One in our party waited ahead of the “Wait Here” line. They moved that person to two different lines and an extra five minute wait. We have no other reason for them being moved, other than being “too anxious”. It wasn’t a problem. There was no extra scrutiny, but a longer wait. As we waited for the x-ray of our belongings after that, they were able to cut the line to us, so it really wasn’t terrible.
Customs into the US
This is where we expected extra scrutiny. However, it was no different that any other international trip that I’ve been on. I’m GOES eligible and used the expedited lines as I normally would. My travel companions were not, and were only five minutes behind me. Same questions you’d get on any international flight.
We found cards at the b&b and hotels to range from 1.50 to 3 CUC for an hour. Once you use them, don’t forget to turn off wifi in case you want to sign in again later. They have no problem letting you stay on longer and having the time expire. Also note that they don’t allow you to finish the left over time on another device. Card is good for one device only. We also tried to set up a Personal Hotspot to allow multiple computers to share, but that didn’t work. It’s “the standard” system, though out Cuba. We found there were times that it wouldn’t get to the sign on page. We opened Safari up and tried that. We re-booted iPhones. Sometimes, it was due to too many people signed on at once and others was that even though you appeared to be successful on the wifi, you weren’t close enough to the hotspot to allow the user sign on screen, which comes after wifi connect, to occur.
US Credit cards still do not work. We pre-paid for all rooms and rental cars to reduce our cash outlay. We converted US dollars to Euro’s prior to the trip. Then Euro’s to CUC in Cuba. Net – we lost $100 through the two transactions. That was way more than the 10% penalty to convert US $’s to CUC in Cuba. We converted directly on the way home.
We used Airbnb for a full apartment for two nights. FANTASTIC. We saw many, many Casa Particular (the sign with the T logo) though out Vieja Habana (Old Havana). Not a chance we would have stayed at any of those or the many, many we saw throughout the country side. Run down terrible looking places. If you don’t know someone that’s stayed there, already, beware. More on our B&B experience in my next entry.
As for the hotels, don’t be looking for a US hotel experience. The hotels we stayed at were older and needed major cosmetic improvements. For the price, I can’t really complain. But buffet meals might have a wide selection, but isn’t always very hot, unless it’s made to order. Rooms were adequate, but in need of repairs. Pools were okay, but also in need of repair. Some of the hotels were abandoned for many years and then “restored” for usage. The level of restoration might be as little as a coat of paint. One of our hotels had signs inside for the original hotel name.
We used GalaHotels to rent hotels in Havana and Varadero. Both looked reputable. Photos and amenities were okay. US credit card processing was good. Price was good. However, they changed one hotel on us two weeks before the trip. A very poor explanation and the email I received didn’t look credible, although the email addresses all pointed to their website vs some bogus site. It was too late for us to argue or re-book elsewhere. It might actually have been given a better hotel, but it didn’t have the same reviews as our original. In hindsight, I might use TripAdvisor next time for the booking. Oh wait, will there be a next time? 🙂
Transportation and Navigation
Lot’s of topics to update here.
Taxi’s in Havana and Varadero
There are many taxi’s from horse drawn carts, Coco cars, 1950’s era cars and modern “yellow cabs”. We found the rates to be reasonable. Ask for the cost to your destination before you get in the vehicle. Also make sure they agree that they know the destination as well. They typically didn’t know our street name, but knew the hotel next to our apartment. We didn’t use any of the Collective Taxi’s as there were always four of us traveling.
Coco taxi – can hold up to three people
Within Havana, the buses are way over crowded and dozen’s of people were waiting to get on board an overly crowed bus. Some people waited hours for buses that seemed to go by regularly, because of the crowding. We didn’t do that.
We rented a car for four days, to get us from Havana to Varadero and back. I wish we could have rented for three days, but there were no vehicles available on a Sunday (they are open 24 hours at the airport), so we got it on Sunday. This is NOT gold service of a US rental car facility. We checked in at 10AM for our car. We got it at 1:30PM. We had to switch airline terminals (Havana 3 to Havana 1), which took over an hour. We rented from Cubanacar, which, like Havanauto, is owned by the government. They shared an office at Terminal 1. There were 4 closed doors and a lot of people waiting. Well, after an hour wait, we were re-directed to the office next door, which was Cubanacar alone. In any case, if you are waiting in line, anywhere, show a native why you are waiting and have them ensure you are in the right place. There is also a 200CUC security deposit and another 95CUC deposit for gas and insurance that you must have cash for at pickup. Several people ahead of us didn’t have that and went to an ATM to Cambio de Moneda to exchange money. 5 to 10 minutes. They wouldn’t wait on another customer until the prior transaction completed. Get some water before you get in line!
The car itself, was nice enough looking. It was an EMGRAND vehicle with 5 speed manual transmission and AC. Other than comfortable seats, the AC was terrible. Plenty of scratches on the car. They had spray painted over some of it. No worries. They did a good job of marking the problem areas so we wouldn’t get charged later. The cigarette lighter didn’t work. We had hoped to use that as a phone charging location. And the transmission was okay, as it shifted easily and correctly, but the car had no power. Most of the speed limit to Varadero was 100 kph/62 mph. The car was lucky to get to 105 kph. On hills, it dropped down to 80 kph/50 mph. We were getting passed by mopeds. It was ridiculous.
We didn’t leave ourselves much time to get to the airport, due to getting lost. (see below). We were able to leave the car at Terminal 3, at the counter we originally waited an hour to get a ride to Terminal 1. They processed the car in less than 5 minutes. We lost the 95CUC deposit because we didn’t take time to fill the gas. But we quickly got the 200CUC security deposit back. We were quite thankful for that. It could have been an additional 1/2 hour to drop off at Terminal 1 and pay for a taxi to get us back to Terminal 3. We didn’t have the time for that.
There are very few street signs. There are very few one way signs. There are no automated railroad crossings. You better stop, look and listen before crossing, as everyone else will. This is even though our B&B host said they’d only seen one train in the last three months.
We had some great maps. Small, laminated tri- or four-fold ones, that were easy to carry. We couldn’t have survived without them. We never could have driven in Havana if we hadn’t walked through it for two days. What looks like an interstate on a map may only be a two lane road, with horse carts, mopeds and pedestrians along the edge, taking up space and going slow.
GPS’ don’t work, as there is no over the air internet through cell phone signal. Download directions onto your phone, prior to using Google Maps. We printed Google Map directions, but it didn’t give you any reference points, other than distance and “turn right”. Not that it mattered as less than 50% of the streets had street signs so you’d know the name.
Example, we were leaving Varadero to go back to Havana. We had driven along the coast on Via Blanca, which was good, but the map implied a southern route was more “interstate like”. We saw a blue directional sign to La Habana, which directed us on a southerly route and a good road. When that road became closer to dirt and in a very congested farm town, we asked for directions. Net: 1:45 hours of driving in the wrong direction. We went almost back to Varadero and went back on Via Blanca. We asked a policeman and he suggested that was 2 lanes in each direction and far faster than the southern route we were looking for.
There are two routes into Jose Marti airport. The shorter one is not marked. We were fortunate that our Airbnb host had picked us up at the airport and driven us back to get the rental car. When we left with the rental car from Terminal 1, we “followed the signs” that got us to the longer route to Habana. As such, when we took the longer route back in, we were familiar with the back roads between the terminals, which again, aren’t well marked. Terminal 3 is for Internationale flights and Terminal 1 is for Nacional flights. Big yellow signs.
Asking People for Directions
We were fortunate to have a person fluent in Spanish and another that had some basic understanding. While driving, we met some wonderful people that were willing to give us directions to the “next major place” and then told us to ask directions to keep going.
While walking, however, we had a different experience. If asking for a restaurant location, multiple times, we were told that that restaurant wasn’t good, but they knew of a better one. Then they’d proceed to go out of their way to walk us there and talk to us about the US and Cuba, along the way. All while being extremely friendly and knowledgable. We learned that they’d get a commission if they took someone to a restaurant or other money making endeavor. We began to stick to our route pretty quickly, as a result.
There are plenty of inexpensive ones to buy. However, we were warned that they were counterfeit. We ended up spending close to 25CUC for very good cigars at a retail shop. Hotel was more expensive for the same thing.
Good news, your US cellphone will work there. Bad news, there is no pre-paid plan. For Verizon, it’s $2.99 per minute to all, about $2.09 a MB (not GB) for data, 50 cents to send an SMS text and 5 cents to receive one. After one day, one in our party got a text that they’d already used $50 in data. A quick call to Verizon to avoid future costs. Their recommendation:
Settings –> Cellular –> Data=off
Settings –> Messages –> iMessage = off
Settings –> Messages –> Send as SMS = On (if you want some contact)
These settings get you to minimal phone usage, in case someone needs to contact you. If you really want to avoid all contact and charges, turn on Airplane Mode.
When on wifi, you might want to change the iMessage back to on. But don’t forget to change it back. For iMessage users that use Email address as the primary contact point, you might be surprised when you open a window to communicate with someone that it won’t go through. Don’t forget to use their mobile phone number, instead, so it goes as an SMS text instead.
We also received 2 to 4 copies of each inbound text. It will be interesting to see if we got billed for each.
Helpful Hint – Shared Notes
Using the Apple Notes iCloud feature that allows sharing of notes across multiple users, we pre-loaded a number of things so we would have them on our iPhones. I am sure there are other mechanisms, like Dropbox or Google Drive that could do the same things.
We loaded notes on Restaurants, Maps, Shopping, Nightlife, Things to do. On each item, ensure it includes an address and phone number. Web links are not helpful at all for additional info. We had to go to those links and cut and paste the content into the notes. Also make sure that you test the sharing on another device before shutting down the source. We created the notes via a Mac computer, but on one note, closed the computer before the sharing completed. That particular note wasn’t accessible on the trip. We just had the title as a tease of our (my) stupidity.
Cutting and pasting a picture of a map (screen shots) including the restaurant or attraction was also extremely valuable. We used some of our paid wifi time to do this once we got into Cuba.Very helpful as we walked around Havana.
What to Do in Cuba
I’ve written a separate entry on what we liked and disliked. If still interested, click here.
I recently saw a blog post by one of the Federal CIO’s. I can’t argue with their observations, though I think we may disagree on how to tackle the problem. That CIO is going to post their direction in future posts. I’m going to take a shot at my own direction in this post.
The following graph demonstrates that the US Government IT is falling behind Fortune 500 firms and way behind internet startups.
Federal CIO study graph
I remember having this debate with an IBM General Manager years ago when he was considering outsourcing some operating system components thinking that all programmers are created equal. There is a huge difference in maintaining a legacy of millions of lines of code vs. starting from scratch with something new. As important, starting over AND maintaining all the rules and regulations of the legacy, is also a very difficult proposition. It takes pre-existing knowledge for success.
This CIO faces a problem that is similar to many other businesses. It’s true for mainframes as it will be for Microsoft Windows and Linux systems in the future. There are millions of lines of “legacy code” in languages that are less popular today than they will be in the future. The inference is to move away from the legacy code toward a modern language where there are more skills available. As a factoid, there are more ARM chips in the market today than Intel chips. There are more applications being developed for iOS and Android than for Microsoft Windows and that’s way more than being developed for mainframes. So that might lead someone to believe that’s the programming model of this generation. And as I’ve said in an earlier post, if your IT career began in the 1990’s and you hated mainframes, you were right….at that time….
But like everything, time changes things. IBM and vendor partners have dramatically changed what the mainframe was into a more modern computing environment. IBM spends over $1B in R&D for each generation of the mainframe that comes out about every two years now. I’m going to park that, for a moment, to go to another topic, that is more relevant to the skills discussion.
Programming is about patterns. Patterns occur at a process level, in languages and in behaviors. There are three broader patterns at work here. Systems of Record, Engagement and Insight. I’ve written about that before, but Record deals with transaction processing, Engagement deals with the end user interface and Insight is about analytics. Most programming being done today is around systems of engagement – taking advantage of enhancements in smart phone, wearable tech (e.g. watches and fitness) and other devices that are the Internet of Things. GPS, accelerometer, touch, voice and biometrics are just a few of the advances that improve the human computer interface. The mainframe has avoided this programming area completely as a native interface. That makes complete sense. Ignored by many, though, is the fact that the mainframe has fully embraced leveraging those capabilities through interoperability and standard formats and protocols. They enable hybrid programming to reach out to those interfaces to simplify the deployment of systems of Record. In addition, they’ve integrated with Systems of Insight to enable real time analytics to be applied to traditional systems of Record to reduce risk.
Another fear raised is that schools no longer teach “mainframe”. Perish the thought. While there are fewer “mainframe” schools than teach commodity system programming, there are a wealth of schools across the world that are part of IBM’s System z Academic Initiative. Checking their website, there are three in Maryland, close to the Federal government and very close to the agency head writing the blog. But you know, “you can’t trust the marketing” materials put out by a vendor. So I went to the Loyola College of Maryland, University of Maryland Eastern Shore (UMES) and Prince George County Community College web sites to see what they said about the IBM Academic Initiative. Honestly, the info I found was from 2011-13, other than Prince George which was up to date. So I reached out to the schools. UMES responded quickly. “First and foremost, I would like to inform you that we are actively involved in the IBM Academic Initiative. Dr. Robert Johnson is the Chair of the Department of Mathematics and Computer Science is the lead person in the initiative. Further, they are currently in the process in moving into our new $100 million Engineering and Aviation Science Building which will significantly enhance our capabilities to support the initiative.” Here’s a brochure for their program.
Most importantly, success is not a two-way street between IBM and the schools. It’s four way, including businesses/agencies and the students. The best schools will work with businesses to provide internships with students PRIOR to graduation. There is generally a very high (close to 50%) success rate in those students choosing full-time employment at the business they did an internship. I strongly encourage any business or agency concerned about future skills deployment to reach out to these schools and work directly with them. Experience shows that you’ll be very pleased with the results. UMES gave me their cell numbers if you’d like to reach out to me for a direct introduction.
Adopt New Technologies and dump the old?
The collective wisdom of the Federal CIO’s seems to point to new technologies as the “future” of programming. The referenced blog points to Uber, Siri and Facebook as examples of such applications and suggests they may be irrelevant in five years. (See Myspace as an example). New technologies grow up in a vacuum. There is no maintenance legacy. It doesn’t mean the legacy can’t work with them, though. A prior blog entry looks at 22 emerging technologies and their relationship to the mainframe and how hybrid computing can solve new business problems.
Let’s consider one of the new, cool tech referenced: Uber. I happen to have a chauffeur’s license (a story for another time) and am very familiar and active with Livery legislation. The Uber mobile application is actually very simple and easy to recreate. What makes them successful is their business model and practices. They hire drivers as contractors, therefore no tax consequences for Uber. They avoid the bureaucracy of Livery laws.
There is a state law that enables the New York City Taxi and Livery Commission (T&LC) to regulate who and what can be operated within the boroughs. This is for the “safety and comfort of passengers”. However, it’s big money. Medallions, per cab, have cost up to $750,000 just to put a car on the street and the T&LC limits the number of medallions. Cars from outside the T&LC are not allowed to make more than one stop in the city. They cannot pick up a passenger at an airport if they dropped them off more than 24 hours ago. The T&LC have 250+ officers in unmarked vehicles that follow and intimidate non-T&LC livery vehicles in the city. I witnessed a stretch limo being impounded by the T&LC when an upstate Livery firm dropped off the passengers returning from a NYC funeral at a NYC restaurant before traveling north. The second stop was illegal. In any event, other states (CT and NJ) got upset with this bureaucracy. They lobbied and a Federal law resulted to allow reciprocal rights to other states to operate without joining the T&LC. But upstate Livery can’t participate. The NY Assembly and Senate have had to modify laws to create T&LC’s in neighboring jurisdictions to allow reciprocal rights in NYC locations. Rockland, Nassau and Westchester counties have T&LC’s now. This is the third year that Dutchess and Ulster have legislation to enable reciprocal rights up for a vote. The NY Assembly has passed their legislation, but the NY Senate hasn’t. Last year, they decided to wait on Dutchess and Ulster until they figured out how to allow Uber and Lyft to operate in NYC exempt from the T&LC bureaucracy. That legislation has now been created and will be voted on soon.
T&LC makes revenue on selling taxi medallions and collecting tax on fares. Uber & Lyft disrupt those economics. The livery vehicles pay $3000 per year for insurance. Uber/Lyft cut deals with insurance companies to lower that to $600/year to make them more competitive. The drivers must also have personal insurance on the cars when a fare isn’t present. Laws are now being enacted to allow “Transportation Network Companies” (TNC as they generically refer to Uber and Lyft) to get “fair access” to markets in NY without this bureaucracy. I’ve developed an app which will qualify the “local” livery company to operate as a TNC to reduce their costs and in turn, reduce the cost to consumers…will the government allow that? Will the Dutchess and Ulster laws pass? This is more about big money, venture capital and paid lobbyist getting to the legislative leaders, than the small livery companies trying to stay in business. We’ll see if the legislation and the bureaucracy will enable the small livery services to morph into a mini-Uber. The legislation enables the Commissioners of Insurance and Motor Vehicles to regulate the “TNC” businesses. The legislation doesn’t prescribe how that will be managed nor how much it will cost. By the way, did you notice that the legislation for Uber includes a lighted icon in the front and rear of the car to identify it? That’s as much for passenger safety as it is to make it easier for the T&LC police to pull over the cars if the legislation doesn’t pass. Not much likelihood of that, though, given the amount of money changing hands in Albany.
Long story short – Uber is more about business processes than it is about new applications.
Past Technology Evolution Examples
Going back to the graph, there is much to learn from prior experiences of the Fortune 500 and government agencies introducing new technology.
Learn from the Fortune 500 – the good:
Benefits processing: Hewitt Assoc and Fidelity continuously advance their capabilities. They provide integration with employer payroll systems. They have up to the minute accuracy of consumer records. They provide immediate access to Accruals and eligibility. They’ve adopted web and mobile technologies as Systems of Engagement, including biometric security authentication.
Claims processing: Travelers Insurance has historically reduced IT and people expense 10% annually while improving response times. Claims agents leverage mobile technology for accidents and disasters as input to “legacy” systems.
Learn from the government – the good:
The FBI and VA leverage mainframe virtualization to avoid IT costs of millions of dollars over commodity systems, while improving security, resilience and service level agreements. They run the same code in a different container with a superior operations model and lower costs.
All of the above use Hybrid technology which includes the mainframe.
Learn from the government – the bad:
Marine Corps – hosted by an IT supplier that gouges them on mainframe costs – three times the amount if they hosted it themselves. The IT supplier takes floor space, energy and cooling costs for an entire data center and only bills to the mainframe users. The IT group claims: Commodity systems wouldn’t be affordable if they were “taxed” with those costs. That’s why understanding the Total Cost of Ownership is a critical success factor when considering mainframe vs. commodity system costs. Unfortunately, regulations are in place that mandate that the Marine Corps use that particular IT Supplier. Other groups have bucked that policy to save money.
US Postal Service was not competitive with package tracking vs UPS and FedEx. They realized they needed to add new applications and wanted modern programming to do it. It included new engagement systems at the delivery vehicles via mobile technology. ….that’s the good. The bad – they spent $100’s of millions on redundant “commodity” IT infrastructure and copied key data and applications from the mainframe in order to host the new applications, while leaving the mainframe running. Testing and benchmarking have demonstrated that adding the new applications to the existing mainframes would have avoided millions in costs and operations complexity, while simplifying the architecture and improving SLA’s. With package shipping volumes increasing annually, they’ve continued to upgrade the mainframe each year. They are just spending too much overall. While they collaborate between the systems by moving data, they could save more if they shared the data in real-time.
Prescription for change
While a prescription for change is forth coming in the CIO’s future blogs, let’s hypothesize some changes for their benefit.
Modernization of the development environment
Rational tools – They move the mainframe application development to commodity systems. This moves 80% of the development off the mainframe to reduce IT costs. They provide tools to modernize and document the “legacy” applications and simplify their maintenance. They provide seamless test to the mainframe and other platforms of deployment choice. One large business has 1000 Java developers for commodity systems, 400 Cobol programmers for the mainframe and 50 developers familiar with Java and Cobol to enable hybrid programming and integration. All use the same Rational development front end. From a skills perspective, the mainframe development can now look and feel exactly the same as development on commodity systems. This eases the skills and knowledge requirements to start.
Cobol Copybooks – the means to define data structures – are now sharable with web services and those services can launch from Cobol. More on that in a moment.
The System z13 server runs dual core 5GHz processors. Benchmarks show that Java runs faster here than any other platform. The video referenced earlier provides specifics. With direct access to databases and files, business applications can have better performance than other architectures. With fault tolerance and an improved hardware and software security architecture, the result is a very price competitive hosting environment for new workloads.
Risk and Fraud analytics
Financial services businesses are doing real-time analytics in the middle of their System of Record transaction programs to assess risk and avoid fraud. Leveraging the Copybook capability, they can call out to leverage the 1000+ processors in the IBM Data Analytics Accelerator (IDAA – formerly Netezza) that have been tied into the mainframe to speed time to resolve.
Callsign – a biometric authentication and fraud prevention technology, can leverage a modern smart phone to identify the owner/user of the device before they actually answer a challenge – which could be a finger print, facial recognition or voice. Using the accelerometer in the phone, the GPS and pressure points on the touch pad, along with historic behavior patterns, Callsign can tell by the way a person is holding a phone if it’s the original user or someone else before offering them the authentication challenge. This type of technology can be used at kiosks in regional/branch offices to enroll users and make sure they are the real person requesting later service. No need for a card. A unique user id is sufficient to provide authentication. True, many low-income users/beneficiaries may not have smart phone capability. Alternative mechanisms can be deployed for challenge/response authentication. But, maybe providing a low-cost device to beneficiaries for this purpose, a more modern version of the “RSA token devices”, might reduce overall costs for low-income users. Watch this space. One of the Callsign customers, a large credit card processing bank, is calling out to Callsign from a “legacy” mainframe transaction program to authenticate that the real customer is at the point of sale or ATM device requesting service. Compare that to an experience I had recently. Visiting 500 miles from home, I went to a big box department store and paid with a valid credit card. Everything was good, but the transaction was denied. I then used a debit card, same bank, same credit card service, but used my pin code. The transaction was approved. As I walked out of the store, I got a call from the credit card provider asking me if I just attempted to use the card. They restored my card to service immediately. Use of the Callsign capability eliminates the human intervention, lowers my embarrassment and speeds transaction processing.
Going a step further, Callsign runs on Amazon Web Services (AWS) or a private cloud today. This is a distributed connection to the transaction systems calling out to it. There are about 15 “risk tests” that can be done, but typically just three can be done and the results fed back to make a risk decision in the time allowed for a transaction to complete. We’ve hypothesized that if Callsign was running on a mainframe, with a memory connection to the transaction programs, that 10 risk tests could be done on the mainframe and maintain the service level agreement of the “legacy” transaction programs. Stay tuned for future updates in this pace.
The NSA has proven that leveraging a Google like search capability can help stop attacks. Why not use web crawling software to look for fraud and overpayments? Leveraging online obituary information, an insurance company or benefits providers could determine if a person has died and no longer eligible for services. In addition, it can predict the services that may be available to the survivors of that person. This can speed up time to deploy payments to their survivors. These web crawlers can feed a data warehouse searching for fraud but also feed real-time systems to avoid fraud for new transactions.
Collaboration is necessary to move forward:
Education: partnerships between vendors, businesses/agencies and schools is necessary to create the next generation of IT professionals (programmers and operations) as well as to update the skills of existing personnel.
Operations: Today, fiefdoms around individual architectures or administrative domains exist that create/foster conflict and drive up IT costs. Not everyone is going to get along. Organizational politics and budgets have as much to do with fiefdoms as anything. Leveraging the Rational developer example, where a small group of people have some hybrid responsibility, can lead to breakthroughs in processing schemes.
Legislation: Where necessary, this can be valuable to enable a leap toward something new that will provide value and reduce costs.
There is no right or perfect answer to any IT decision. As the saying goes and leading to an unintended consequence: “Throwing the baby out with the bathwater” isn’t necessarily a good approach. Leveraging a hybrid computing, operational and development environment can make a large shift toward leveraging “modern” application models. Happy programming!