Cloud Power: Operations costs are the Achilles’ heel of NoSQL

Check out the latest post at my Cloud Power blog at Computerworld:

Companies interested in adopting NoSQL should consider their options carefully. The vast majority of database use cases do not need massive horizontal scalability. Most applications could be better off with traditional SQL databases. In the cloud, there are NoSQL alternatives that cost less and are easier to maintain.

via Operations costs are the Achilles’ heel of NoSQL | Computerworld.

Cloud Power: IT departments must transform in the face of the cloud revolution

In case anyone’s been wondering why there hasn’t been an update to this blog in a couple of weeks — well, I am now part of the Computerworld contributor network with a blog called “Cloud Power“.

My first post is on the topic of Shadow IT  :

Cloud computing democratizes developer and end user productivity at the expense of transparency and IT control. Since developers and users are able to provision and utilize resources as needed, it is easy for costs, overall architecture and security to get out of control. Rather than getting in the way of productivity, however, the IT departments must evolve their role from that of the gatekeepers into that of enablers.

Enjoy the read and join the conversation!

Banking Technology is in Dire Need of Standartization and Openness

Old Bank Photo credit Toby Dickens
Old Bank
Photo credit Toby Dickens

A few weeks ago Investors Bank in New Jersey overhauled their systems. As a result Mint became incompatible with Investors and Investors customers could no longer view their account in Mint. There is anecdotal evidence1 that Mint uses the Yodlee platform2 for the integration. As it turns out, there is no standard mechanism by which external applications can work with banks. Yodlee’s own page states:

Through a proprietary system of direct data access and HTML parsing, Yodlee delivers financial data from more than 14,000 sources, and growing.

While the technology world is moving towards open APIs and standard authentication protocols3 the banking industry continues to rely on proprietary systems and HTML screen scraping. It seems that even using Yodlee platform it is not possible to integrate with banks in any standard way. Each time a bank updates their systems a team of engineers at Intuit must update integration scripts to ensure their customers can continue to use Mint with that bank4:

When a financial institution updates their system, our engineers have to rewrite the script on our end to match so that we can continue supporting them. Typically, they are notified when this is going to happen and can get it updated pretty quickly. However, please open a ticket by filling out our Contact Mint form to make sure this is on their radar and they can get the script updated as soon as possible.

The way Mint integrates with banks is by asking users to enter and store their bank credentials. Mint expects us to trust their security5. The technology industry, however, has long established a protocol by which an application (like Mint) needing access to an outside resource (a user’s bank account) does not need to capture user’s credentials. It is called OAuth6.

Had banks implemented OAuth, mint would use the protocol to obtain an authorization from the user to act upon the bank’s API on behalf of the user. In the event of a security breach at Mint it would be possible for the banks to invalidate all tokens — and disable all further access by Mint. Users would gain control over which applications they want to access their data and which they do not.

In 2015 there is no need for HTML screen scraping or proprietary technologies. Would Yodlee platform even be around if the banks used OAuth and standard API7 ? This is an industry that is in dire need of innovation. Banks need to learn how to recruit and retain top talent from the technology companies, not the other way around. They need to look beyond their traditional well accepted consulting vendors and service providers and think outside the box — especially considering the fact that the technology challenges they face have already been solved by others.

I Stand With Ahmed

This week a precocious 14-year old immigrant Ahmed Mohamed wanted to impress his teachers with a clock he made at home. He built it into one of those pencil boxes you buy at a craft store that look like a small brief case. The teachers and school officials thought it looked suspicious and called the police. The police proceeded to arrest him as a terrorism suspect1.

This is a technology blog and so I won’t get into the topics of politics, racism, and terrorism. Let’s even set aside the seemingly incompetent reaction of Irving, TX law enforcement who had not evacuated the school. Instead I am going to focus on the topic of STEM education in the United States.

My 8 year old daughter building an Arduino LCD circuit
My 8 year old daughter building an Arduino LCD circuit

It just so happened that a few days prior to this incident my 8 year old daughter asked if she can bring the Arduino LCD circuit I had built with her to school to show her friends and teachers. I was not even thinking that an elementary school teacher may think a circuit with batteries, wires and a display is a bomb and it may result in her arrest.

To tell the sorry state of American STEM education all one needs to do is take a tour of top engineering universities and visit science and engineering classrooms. A keen observer will find that the majority of students are immigrants. These students have multiple advantages over American students — they come from cultures that value knowledge and education, families that invest in their childrens future, and teachers who can a tell a bomb from a clock.

Of course, what starts in universities transfers to workplaces. A visit to any software company or even an IT department just about anywhere will reveal that the majority of developers are immigrants as well. They come from India, China, Ukraine, Belarus, Russia, and elsewhere in Asia and Europe.

Meanwhile, American politicians draw crowds of people at campaign rallies fanning the flames of fear over American jobs2. The reality, however, is that a much bigger threat to the future of American middle class jobs starts in schools. When teachers, school, and law enforcement officials can’t tell the difference between an explosive and a homemade clock — how can American kids look up to them ?

Setting Up Cross-Region Replication of AWS RDS for PostgreSQL

As of today AWS RDS for PostgreSQL1 does not offer cross-region replication. Short of switching to one of the RDS offerings that do support it, there is a few options to consider.

1. Custom Configured EC2 Instances with Master-Slave Replication

Master/slave light rail
Master/slave light rail

This setup sacrifices the benefits of AWS RDS service in exchange for greater control over replication settings. In this setup, one region hosts a master PostgreSQL host, and another region hosts a slave which can also act as a read-replica2.


Greater control over replication settings.


  • Give up all the advantages of running in AWS RDS environment.
  • Writes can only be performed in the master region.

2. Software-defined Two-phase Commit

Commitment Photo credit: Ed Schipul
Photo credit: Ed Schipul

In this setup there are two independent AWS RDS instances. The application, however, utilizes a two-phase commit protocol3 to guarantee that all writes make it into both databases in a transactional fashion.


  • Simple configuration
  • Does not sacrifice any of the AWS RDS advantages


  • Responsibility for ensuring that writes make it into all regions fall onto the application itself.
  • Increased application code complexity.
  • Write performance is sacrificed since all regional databases must participate synchronously.

3. Asynchronous Writers

Fanning Out Photo credit: Tim Haynes
Fanning Out
Photo credit: Tim Haynes

In this approach each region hosts an asynchronous writer that listens on an SQS queue4. All writes are published on the SNS topic that is configured with all regional writer queues as subscriptions5. When the application running in any of the regions wants to write into the database it publishes a message on this SNS topic which then fans it out to all of the regional SQS queues.


  • Simple configuration
  • Does not sacrifice any of the AWS RDS advantages
  • Does not sacrifice write performance


  • Subject to software bugs
  • Subject to SNS and SQS bugs and outages
  • No guarantee of consistency
  • Requires a mechanism for periodically reconciling differences between regions

Top Ten Differences Between ActiveMQ and Amazon SQS

Taxi queue at LaGuardia Photo credit: Scott Beale / Laughing Squid
Taxi queue at LaGuardia
Photo credit: Scott Beale / Laughing Squid

1. Persistence and Durability


Depending on the configuration ActiveMQ can maintain a message journal1. Each message is first written into a journal before being shipped to consumers. Ultimately, the number of messages that can be persisted is constrained by the available disk capacity.


Amazon SQS stores messages in a distributed storage across all availability zones in a given region2. Each message size can be up to 256KB and SQS can store an unlimited number of messages across unlimited number of queues3.

2. Redundancy


ActiveMQ offers a number of different configuration options for clustering4:

  • Broker Clusters and Networks of Brokers: this architecture is most appropriate for distributed networks of brokers. Producers on each broker can reach consumers across the entire cluster. This is most appropriate for a use case such as delivering market data to all consumers across the entire network (JMS topics). This is not exactly a redundant configuration – failure of a single broker results in message loss on that broker.
  • Master-Slave : In this configuration two or more ActiveMQ brokers use some sort of a shared5 storage for the journal. Prior to ActiveMQ 5.9 one had to relied either on a shared file system such as SAN or on an SQL database – which simply shifted the replication responsibility to a different technology. Starting with ActiveMQ 5.9 there is an option to use Replicated LevelDB with Zookeeper6.


SQS stores messages in redudant storage across all availability zones in a given region. To achieve high levels of redundancy and guarantee that no message is ever lost it relaxes some of the properties of a queueing system7. What that means is that on rare occasions messages may arrive out of order, and same message may be delivered more than once.

3. Graceful Failure


In a master-slave8 configuration all clients failover to the next available slave and continue processing messages. In any other configuration, all processing stops until the client is able to reconnect to its broker.

In the event of high memory, temp storage, or jounal space usage ActiveMQ can pause producers until the space frees up. This creates a potential for a deadlock situation where some consumers also act as publishers and become unable to publish or consume messages. There is a risk of the entire system locking up until space is freed up or configuration is changed.


When your application attempts to retrieve messages from a queue SQS picks a subset of all servers and returns messages from those servers. What that means is that if for some reason a server was unavailable a message may not get retrieved – but will on subsequent requests. This is mitigated to a certain extent by use of long polling9.

4. Message Order and Delivery Guarantee


Messages are delivered in the order they are sent10. When there are multiple consumers on the same queue some of the order may be lost – however, that is the case with any queue that has multiple consumers and it is exacerbated by clustering configurations.


In order to achieve high levels of scalability and redundancy SQS relaxes some of the guarantees of a traditional queuing system. On rare occasions messages may be delivered out of order and more than once, but they will get delivered and no message will be lost. Applications sensitive to duplicated or out-of-order processing need to implement logic to cover these scenarios11.

5. Monitoring and Utility API

This may seem off topic but I do find it necessary to mention. It is often useful, from application standpoint, to perform various utility functions against queues. An application may measure the rate of dequeuing, calculate number of pending messages, and self-optimize.

JMS does not offer API to retrieve this information. ActiveMQ does expose some of this via JMX, however12. Similarly, SQS offers metrics and utility API as part of the SDK.

6. Standards Compliance


ActiveMQ conforms to the JMS API specification in the Java universe and has drivers for other platforms and API specifications.


SQS uses HTTP REST protocol and a proprietary SDK. However, Amazon does offer a JMS implementation of the SQS SDK13.

7. Push Messages as They Become Available


The default ActiveMQ protocol is based on a socket connection that allows messages to get pushed to the consumer as soon as they are published. With JMS one can implement MessageListener14 interface and receive messages as they arrive.


SQS does not natively support push. One has to poll to retrieve messages. This is a minor inconvenience since Amazon provides both long polling and a JMS implementation. Various approaches exist to mimic the push behavior including one that I described in my post on “Guaranteeing Delivery of Messages with Amazon SQS.”15

8. Scalability and Performance


ActiveMQ can handle tens of thousands of messages per second on a single broker16. There is a great deal of tuning that affects ActiveMQ performance including the host computer capacity, network topology, etc. Scalability is achieved either vertically by upgrading broker hardware or horizontally by expanding the broker cluster.


SQS does not return from a SendMessage request unless the message has been successfully store and as a result it has a request-response latency of around 20ms. At first glance it may mean that it cannot handle more than a few hundred messages per second.

However, when dealing with a distributed queue like SQS one has to distinguish between latency and throughout17. SQS scales horizontally. By using multiple threads it is possible to increase message throughput almost indefinitely.

9. Setup, Operations and Support


ActiveMQ is just like any other software that one has to install, configure, monitor and maintain. Configuring and tuning ActiveMQ requires thorough understanding of hundreds of different settings18. ActiveMQ itself is written in Java so understanding of Java topics like memory management and garbage collection is helpful.


As long as you are operating in the AWS environment there is nothing to configure, install or maintain. SQS is a completely managed service.

10. Costs


ActiveMQ needs hosts to run on and storage it can use. Someone has to support and maintain it. The costs of ActiveMQ are a function of resources it needs to run and time it takes to tune, configure and maintain it. These costs are still present during periods of low utilization since it doesn’t scale automatically.


SQS is priced as a function of number of requests and data transfer. You are only charged for what you consume, so during periods of low utilization the costs are lower.


The discussion in this post boils down to the choice between a fully managed cloud service and an installable software product, just like DynamoDB vs Cassandra19. A managed service simplifies development and maintenance at the expense of standards compliance and customization options.

We Live in a Mobile Device Notification Hell

Notification Hell
Notification Hell

On a hot Sunday afternoon I found myself walking around Menlo Park Mall in central NJ with my wife and kids. My phone vibrated because someone’s automatic spambot just faved a dozen of my photos on Flickr1. Bitstrips app wanted to let me know that I have new Bitstrips waiting for me. 10App demanded my attention reminding me to make a YouTube video of what my kids did today.

As we walked past Verizon store, my phone got all excited telling me about all the things I can buy there. Apple Store wanted to remind me I have my order waiting for pick-up, even though I picked it up a week ago.

Flipboard decided to notify me that a barely dressed coffee aficionado interior decorator started following me. I have hundreds of messages unread in my personal email account and dozens of LinkedIn notifications of recruiters telling me about “Urgent Java openings” that have nothing to do with my career goals.

When I got back home my iPad’s screen was filled with the same exact notifications that my iPhone told me about, as if iPad is unaware it is owned by the same person and that I already acknowledged them. To make the matters worse my MacBook’s notification screen was repeating them as well.

We live in a notification hell world of smartphones, and every year it is getting worse. Our presumably smart devices are incapable of differentiating between what is important and what is not. The social sharing apps like Facebook, Twitter, and Instagram want our constant attention. Flickr is now a spam-bot haven — any time I post a picture, any hour of the day, it is immediately favorited by the same 3 people who have millions of favorites in their photostreams.

No wonder I have no desire to buy a smart-watch2 and I keep my iPhone permanently in a “Do Not Disturb” mode. Why would I want to add a yet another device that I have on me that will constantly demand my attention ?

Buried in their smartphones

While I miss the days of simple flip-phones, I can’t deny the convenience of smart mobile devices. They allow us to work where and when we want. They allow us to get the best price for products we shop for. Yet, I would love nothing more than to stop all the meaningless blinking, beeping and flashing.

We need intelligence built into mobile push notifications. While it is possible to selectively enable or disable notifications by the app, it is simply not enough. When I see a notification I want to swipe it and say “It’s not important” and have my device learn over time and stop alerting me of it3. This learning is then propagated to all of my devices.

Once I acknowledged a notification there is no need for my other devices to tell me about it again. There is nothing stopping my MacBook, iPhone and iPad from knowing that I already read my brother’s Facebook update. They can decrement the notification counters and remove that notification from their respective screens.

There used to be a joke in the software engineering circles that a software platform reaches the end of its natural lifecycle when it becomes capable of browsing the web. In 2015 it seems that any app loses its usefulness the moment it allows social sharing and public APIs. Once social sharing is enabled and public APIs are published the app becomes a medium for spam. Consider all the outfits that let you “buy” Twitter, Instagram, Flickr or Facebook followers.

Social Media
Social Media

I used to love Flipboard and used it daily to read the news. Then one day Flipboard allowed “likes” and “follows”. Within days I went from zero followers to a few dozen followers, all of which are skinny women calling themselves “internet mavens”, “social media aficionados” and “interior decorators.” Somehow they were all interested in Big Data, international politics, and stock market investments. I uninstalled Flipboard until I read somewhere that they started allowing private profiles that one has to opt-in.

It is not complicated for social media platforms to tell who is a bot and who is not. On Flickr, for example, an account with a million favorites but only a couple hundred photos that haven’t been updated in a couple of years is a spam bot4. These platforms can impose API limits — it is simply not humanly possible for someone to have a million favorite photos on Flickr, for instance.

Vast majority of us are not doctors5, military, police or firefighters — we have no real work emergencies. Most of us do not deal with life and death situations as part of our jobs. In software engineering what we typically call emergencies are self-inflicted manufactured crises. And yet, with proliferation of smart mobile devices we are expected to be constantly in contact with our work.

We need enterprise apps on our devices to know what’s important and to learn what is not. Enterprise apps should not be constantly notifying us of “work” we would rather not be doing on our spare time. Instead, they should be reminding us of our goals and helping us succeed.

  2. Why I am not getting an Apple Watch 
  3. I am thinking something along the lines of a Bayesian-network based spam filter. 
  4. It is disappointing to see some well-known photographers utilize the services of spammers. 
  5. Software Engineers Are Not Doctors