Smart IT Departments Productize Business API and Take Ownership of Data Governance

Satya Nadella explained Microsoft’s “secret” weapon against AWS and Google:

To me what matters is having the right mix of SaaS value. I don’t think of my server business as somehow “old school” or “legacy.” I actually think of the server as the edge of my cloud.

We now have the ability to tie together the cloud and the server. That is a very unique capability that we have. So who am I competing with? Amazon has no capability to compete there. They don’t have a server. Nor does Google. Oracle doesn’t have the equivalent capability. So those are the places where we want to really excel.

The reality is that whether you have some sort of a server on-premises or not shouldn’t matter and that is why Google and Amazon are not concerned in the long term – while Microsoft will continue to be a follower in the cloud arena rather than a leader.

Let’s consider what public cloud like AWS has done for the software industry: it dropped the cost of entry for a startup to near zero. Whereas in the past a startup would need to get redundant enterprise-grade Internet connection and build out a server infrastructure, today all they need to do is going to their AWS dashboard and provision a server. The point is, Googles, Amazons of tomorrow are not built on-premises. Startups of today will be the dominant players of tomorrow and they are built in AWS and Google without a care in the world for on-premise IT.

Smart enterprise does not rely on any particular server. I’ve long been advising my employers and customers to not use Java EE servers like IIS, WebLogic or Glassfish, for example, and instead rely on lightweight platforms (Spring and Jetty for Java, Node.js, etc.) Smart enterprises build out enterprise API that make the location of their applications (on-premise or in the cloud) irrelevant to the business. Smart IT departments develop data governannce policies that improve insights while decentralizing data.

The cloud shifted the center of technology management and thought leadership away from enterprise IT departments and CTOs down to individual teams. A team armed with a budget no longer needs to go through red tape and beg their IT department to find a place where to run their applications in a scalable fashion. Just like BYOD disrupted enterprise mobility so does “Bring your own Salesforce”, “Bring your own AWS” and “Bring your own Heroku.” Enterprise IT can help make themseles relevant by not restricting where the applications are hosted and instead offering secure enterprise API accessible as described above, along with data governance and best practice procedures.

An enterprise that owns and productizes their business API and has a sound data governance is not beholden to any particular cloud vendor – not even Microsoft. Traditional vendors will continue to sell their hybrid on-premise/cloud products, but the reality is that they only kick the can down the road and further entrench their influence in your organization. Own your API and data governance and set your enterprise free from the shackles of enterprise IT vendors!

Guaranteeing Delivery of Messages with AWS SQS

I wanted to scratch an itch and get feedback from the open-source community. So, I put together a little Github project that I like to call SQS-RetryQueue.

Amazon SQS can be utilized to guarantee delivery and processing of messages. This project serves the following purposes:

  1. Demonstrate an example of using AWS SQS with Java to send and receive messages.
  2. Provide an algorithm for retrying failed deliveries.
  3. Provide an approach to keeping SQS costs to a minimum while maintaining real-time processing of messages.
  4. Seek feedback on the approach from the open-source community

AWS SQS is priced by request. One of the goals should be to minimize costs.

Processing of messages can happen on either the server that sent the message or any other server subscribing to this queue. The goal is to begin processing messages as soon as possible.

Each receiver thread acts as follows:

wait on the monitor object for up to visibilityTimeout
while there are messages on the queue:
    receive message
    try processing message
    if processing was successful, delete the message

Sending a message then involves the following:

send the message
notify all receivers waiting on the monitor object

I wanted to get this out of the way for some time. So, here. it is! Any feedback is greatly appreciated.

We Need a Cloud Version of Cassandra

Google recently launched Cloud BigTable – a cloud NoSQL service that is compatible with Apache HBase API. What this means is that an existing ecosystem of Hadoop applications is immediately binary-compatible with this new service and it doesn’t require any changes to the API.

Google is not only marketing this as an alternative to Hadoop, but it is also marketing it as an alternative to Cassandra.

Now, I’ve written in the past that what we need is a proper cloud version of Cassandra. Datastax is a rising star but the fact that none of the cloud providers have a managed, low cost, automatically scaled cloud offering that is compatible with Cassandra API is going to be their downfall.

What burnt me with Cassandra is the fact that as you add storage and capacity by adding more nodes you increase devops costs. What I would like to see from Datastax is not a partnership with HP whose goal ultimately is to sell more hardware and services to on-premise data centers. Instead, we need Datastax to form a strategic partnership with Amazon AWS and offer a zero-maintanenance, zero-initial investment, zero-devops, auto-scaled and fully managed binary API compatible Cassandra offering in AWS.

The Clarkson School Class of 2015 Commencement

Last weekend, on May 2nd, I had an opportunity to come to Clarkson University and give an alumni address at the Clarkson School Class of 2015 Commencement.

The best way to describe Clarkson School is that it is bridging year program where talented high school students get to leave high school a year or two earlier and attend first year of college at Clarkson University. They live in their own dorms and participate in activites under more supervision than typical freshmen would. It is a somewhat unique program and is a hidden gem.

I was in the Clarkson School in the 1996-1997 and it was one of those fundamental turning points that shaped the rest of my life. I am deeply thankful to the Clarkson School for the opportunity and it was a great honor and an amazing experience for me to come speak to the Clarkson School class of 2015.

Full text of the speech is here and you can watch the video below:

The Clarkson School Class of 2015 Commencement from Oleg Dulin on Vimeo.

Why I am not Getting an Apple Watch For Now: Or Ever

A friend of mine received his Apple Watch recently and I had a chance to play with it. In short, I am not swayed.
I am a fan of wrist watches. Personal timepieces have been marvels of engineering for centuries. I have an Oris Complication automatic, a couple of Citizen Eco-Drive wrist watches, and a $25 Casio sports watch. The one I wear 90% of the time is the Citizen Eco-Drive world traveler watch with atomic clock.

With the exception of the $25 Casio sports watch whose battery will die any day now, all of my watches can last a lifetime without looking outdated. The Oris requires regular maintenance due to its mechanical nature, but the Citizens don’t even require charging, ever. All of them are waterproof to 20 BAR.

While smartwatches are a huge step above the dorky Google Glass as far as wearable computing goes, they are far from convincing me to purchase any of them – Apple, Android or Pebble.

One issue is battery life. It means dragging extra cables and extra chargers with me when I travel. It means that on a 10 hour flight to Europe I have to worry about a yet another device dying on me. From what I gather, even the app to control a GoPro using the Apple Watch requires the phone to be paired with the GoPro first – so now I am draining batteries on three devices, instead of two.

Second issue is that I just don’t get the point of having what effectively is a smaller external screen for a device that is already in my pocket. To take advantage of any of the functionality the Watch needs to be in the vicinity of the smartphone. This is a make it or break it for me.

I would gladly give up my smartphone and exchange it for a smart watch with its own cell chip. I imagine a watch, just like Apple Watch, that has its own cellular and wifi chip that can work completely independently of the phone. I can use it as a hotspot for my laptop or my iPad. I can use it to make phone calls – using either speakerphone or a headset. In fact, the wrist band can have a speaker on one side and a microphone on the other so when you unclasp it from your wrist the watch works as a flip phone. When I get into my car, the watch pairs with my car’s bluetooth system and let’s me make phone calls. All of this without having a smartphone, or yet another device that requires cables and charging.

Third issue is that I happen to like the classic looks of the watches that I own already, and Apple Watch (and all other smartwatches for that matter) is quite ugly. Walk into any watch store and you will see a huge variety of shapes and sizes. To imagine a world where everyone wears an ugly smartwatch is to imagine an Orwellian or a Brave New World reality where everyone thinks and acts exactly the same.

Fourth issue revolves around notifications. Today, when someone messages me on Facebook my phone vibrates and shows notifications. My iPad, which is at home, also gets a notification. So does my computer. Even if I read them on my phone, the same exact notifications are waiting for me when I come home. To add a yet another device that will be beeping and blinking any time my mother clicks “Like” on Facebook, or any time my boss has a question would send me into therapy.

I am sorry, Apple and Google, but for now I am simply not interested. Make me a product whose battery lasts for at least a week, that can be used without also owning a smartphone, that won’t become outdated in a year, and I will take a look again.

My Brief Affair With Android

As a software engineer I like to experiment with different technologies and step outside of my comfort zone once in awhile. Having used iOS devices for a very long time, sometime last year I bought myself a used Samsung Galaxy Note 3. I can now confidently say that whoever thinks Android is better than iOS must also be the kind of person who thinks Windows 98 is better than Mac OS X.

Allow me to explain.

When my Samsung Galaxy Note 3 running Android 4.4 Kit Kat arrived the first that I was greeted with was a bunch of pre-installed apps I did not ask for and did not need, one of them unfortunately named “ISIS Wallet.” Verizon and Samsung preinstall a bunch of crappy apps that you cannot uninstall just like Acer and Dell used to (and still do) bundle unnecessary apps with their Windows laptops and desktops. Fine, we can move past that.

The device had a stylus. Woohoo! Exciting! I never used it. Ever.

The next thing I observed was absolutely horrendous notifications. If you lock the screen with a passcode you don’t see your notifications unless you unlock it, meaning you can’t at a glance tell what’s going on. App icons don’t show little bubbles like iOS does and across the top there is an incomprehensible bar of icons and indicators that gets filled up with meaningless nonsense. Oh, and the LED on the front of the device would light up in christmas light colors when there were pending notifications, but the colors mean nothing.

So that was Android 4.4 Kit Kat and I honestly thought that the device was so old it couldn’t support Android 5.0 Lollipop. I was fine with that. A couple of days ago, however, I got a notification saying something like “Samsung Has Prepared Android 5.0 Lollipop Update for Your Device. It has exciting new features.” So I said “Give it to me.”

After about 15-30 minutes of updating the device, I was greeted with an error message saying that my device is not compatible with Google Play Services. Furthermore, my Google Calendar stopped syncing.

Sorry, Google, I am not interested. I want my stuff to work. I have no desire in debugging what really should have been an automated process. If my device was not compatible with Google Play Services and installing Lollipop was going to break some things that make smartphones smart, it should not have installed Lollipop.

In any case, I have no desire to mess with the settings, to install and uninstall software, and so on. All I know is that my iOS devices always worked after updates. My contract with Verizon is up, I am going over there and will get a new iPhone this weekend.

Apple is (or was) the Biggest User of Apache Cassandra

One thing I did not realize about Cassandra is that Apple is (or was) one of the biggest Cassandra users out there:

Word in Goldmacher’s circles is that Apple will be “replacing” its huge Cassandra noSQL implementation with FoundationDB. Apple uses Cassandra for “iMessage, iTunes passwords, a bunch of stuff,” he says.

In fact, Apple is touted as having one of the largest production deployments of Cassandra of all, with over 75,000 nodes storing over 10 petabytes of data. Cassandra is a free and open source database with a commercial version offered by DataStax

The article further states that FoundationDB can operate on cheaper hardware, less nodes, and faster. It states that Apple could reduce their cluster size by 5-10%.

5-10% off of a cluster that size is not something to be sneezed at. We are talking upwards of 7500 servers and millions of dollars in savings in hardware and even more devops costs.

Since RAM is the new disk, disk is the new tape an in-memory data store backed by a disk is going to support reads that are orders of magnitude faster than a data store like Cassandra that uses disk as a primary storage mechanism. For example, Redis has a data model that is similar to Cassandra but it is entirely in-memory.

Of course, it all depends on your requirements. If your needs are to accumulate massive amounts of information that is queried infrequently or in off-peak batches, then Cassandra is very appropriate. But if you require consistent performance for both reads and writes you should look elsewhere.