In memory of Ed Yourdon

Ed Yourdon
Photo credit Ed Yourdon

Every generation assumes, in its youth, that it is immortal and omnipotent. And every generation of children ignores the advice of its parents, believing that their circumastances are so new and different that the lessons of their parent’s lives simply wouldn’t apply. On the surface, this seems to be true in computer field, too: Why would today’s young Java programmer believe there is anything to be learned from experiences of a mainframe COBOL programmer ?

Ironically, this attitude of generational arrogance is part of the basis for my optimism for the American software industry. If today’s generation of software developers followed in the footsteps of their elders and used the same kind of technology and practices, they would be subject to the same kind of crushing competitive pressures that the older generation is facing around the world. But they don’t – they prefer, instead, to leapfrog over the older technologies and plunge into something new. And in most cases, the older generation encourages them to do so; even if we’re trapped in our old paradigms and technologies, we have enough sense to encourage our children to try something newer.

– Ed Yourdon, “Past, Present and Future”, in Rise and Resurrection of the American Programmer, Yourdon Press, 1996

At every stage in my professional life I met and got to know people I consider mentors and role models. Some were pioneering technologists who pushed the boundaries of the software industry. Others were professors, coworkers and leaders. Each person I admire and respect in a different way to this day. I would like to talk about one in particular who has been crucial to my growth as a professional and a human being.

When I was in college in the late 1990s I came upon two books: “Decline and Fall of the American Programmer” and “Rise and Resurrection of the American Programmer” by Ed Yourdon. The first book spelled doom and gloom for the American Programmers who were going to get replaced by cheaper counterparts in India, Russia, Philippines, etc. The second book revisited some of the predictions based on the changes that the software industry has undergone in the years between the books. Both books were incredibly thought provoking. To this day they occupy a prominent spot on the bookshelf in my home.

Computers are remarkable devices that only humans, as species of this planet, could conjure up. Software development is the most cognitively complex task humanity has ever set out to pursue. In computer science and software engineering we work with things that we ourselves build. If a computer fails, one can say that it failed because the engineer who built it did not know what they were doing.

Building on the work of Aristotle, Alexander of Aprodisias developed a notion of a stochastic art:

Given materials, tools, and other conditions, carpentry, e.g., can produce houses by following a series of steps each of which is effective in a determinate way. However, medicine does not always cure and certainly does not cure with the reliability that carpentry produces houses. Even though medicine tries everything in its power, chance can intervene so that it does not achieve its goal, the curing of the patient. When carpentry, by contrast, tries everything in its power, it achieves its goal. Failure here is the result not of chance but of error in executing the technê, as Alexander says in Quaestiones (Quaestio 2.16, 10-25). To mark the difference between these two kinds of technê, Alexander says that the task (ergon) of medicine is to try everything possible to achieve its goal (telos); but achieving its goal is not (totally) within the power of medicine. He calls stochastic, then, the sorts of technê whose task is to try everything possible to achieve their goal, the realization of the goal being subject to chance.

In software, everything is in the power of the engineer to produce a quality product. Software either works and serves the needs of the users or it does not. In photography, however, we deal with things not of our own making. Many software engineers gravitate towards photography as a way to settle the mind, to unwind, and to work with things and events we have no control over. The desire to be in the moment without being able to “debug” and retry is uniquely human.

As I entered the professional world of software engineering I too developed a hobby in photography. Sometime in 2005 I was looking through Flickr groups for ideas and techniques and came upon Ed Yourdon’s Flickr account. The same computer scientist whom I admired in college turned out to also be a prolific street photographer.

As a photographer he was no William Klein or Brandon Stanton, but he added his own unique flair to photography. Photography was like a diary for him. By following Ed on Flickr one not only got to know his beloved New York City but also Ed as a person. His NYC street photos such as these inspired my own attempts at street photography.

Thanks to social media I became friends with Ed and got to know him closer. As we both developed our hobbies we exchanged ideas. Checking up on his photography became part of my morning routine. He commented on my photos and gave me tips. He accepted my ideas and tried new techniques. As inductees into the Computer Science Hall of Fame go, he was open, kind and friendly.

Moses Ben Maimon (aka Maimonides) lived over 800 years ago. With his studies and writings he influenced thinkers of his time and his work is studied the world over even today. We remember him today because of the things he wrote, and what was written by others about him.

What Ed was striving to accomplish throughout his professional career and his hobbies was to make the world a better place and to leave a footprint. He published dozens of books and hundreds of articles explaining complex topics to the rest of us. He posted thousands of photos to his Flickr account. It is nearly impossible to search for an  NYC street photo on Flickr and not stumble upon Ed Yourdon’s pictures. His photos, which he gave away via creative commons, have been used in thousands of blog posts and articles.

In the past few years, when someone asked me that cliche interview question “Oleg, where do you see yourself in the future?” I would respond “Do you know Ed Yourdon ? I want to be like him!” On occasion he would drop me an email encouraging me to develop professionally. It was by following his example and tips that I’ve improved my writing and became a contributing blogger at “Computerworld”.

He posted to his Flickr account almost daily. It was rare for him to take more than a few days away from photographing and posting. His last post was on December 24th, 2015. After a few of weeks of not seeing updates or hearing from him I suspected something was wrong. On Thursday morning, January 21st, I saw this in my Flickr notifications:

Sadly, Ed Yourdon died on January 20, 2016 as a result of complications from a blood infection. Photography was one of his great passions in life. He greatly enjoyed the camaraderie he found via Flickr.

Ed Yourdon is a Maimonides of our generation. His work in computer science and software engineering shaped our industry at a time when it needed structure. His photography gave us a glimpse into his life and his values. The world is in a better place now because of Ed. He will be greatly missed.

Our civilization has a single point of failure

2015-10-06 12.34.55-1

When I was in Las Vegas in October of 2015 for AWS re:Invent I took some time to go for a walk around the Venetian. I came upon Bauman Rare Books. In the store, I saw a copy of Saint Exupery’s “Le Petit Price.” It was a number 231 out of 260 copies signed by the author. It was selling for $23000.

When Amazon Kindle first came out in 2008 I was one of the early adopters. The device and the format proved perfect for me. The ability to impulse-purchase an electronic book anywhere I am and carry hundreds of books in one thin device allowed me to read dozens of books in the first year I got the Kindle. I can change the font size and read the books even when my eyes are tired. I don’t get motion sick reading Kindle on the bus, something that routinely happens to me with printed books.

None of those books are special, in any way. They all look the same on my Kindle, just a row in the catalog. There is nothing that makes them “limited edition.” None of them will ever be signed by an author. None of those electronic books will ever find themselves in a rare book store. The Kindle itself is unlikely to ever find itself on a collector’s shelf.

To make the matters more precarious, when you buy an ebook you don’t actually own it. In 2009, Amazon remotely wiped copies of Orwell’s “1984” from the Kindle devices. There are articles and guides out there on how to protect your ebooks from the same fate, such as “DRM be damned: How to protect your Amazon e-books from being deleted” on the.

On the opposite side of the debate, there are opinions justifying Amazon’s actions and their right to delete e-books from customer devices. When you pay for an ebook, you pay for a license to read it on your Kindle. This is analogous to how software is bought. When you pay for an app, you do not own it — you purchase a license to use it and it can be revoked at any time, for any reason, with little warning. The same can, did, and will happen with e-books and electronically purchased music.

Despite all that, I have long since stopped buying printed books. The convenience outweighs the perception of impermanence. If the book is valuable or special, then I buy both — the e-book for the convenience, and the printed version for the long term value. I can literally count with the fingers of one hand the number of printed versions of e-books that I purchased since getting a kindle.

If the power outages following the hurricane Sandy in the October of 2012 were a couple of days longer I would have had nothing to read. Imagine for a moment what would happen to the modern civilization in the event of a catastrophic solar flare that would at the very least suspend modern civilization, if not completely up-end it. If the power is out for months, and electronic device memories wiped clear, what will the humanity refer to for knowledge and guidance? What will happen to our family photographs, our music collections, and our Kindles?

People no longer collect music, they subscribe to it. We post thousands of photographs to Instagram and Flickr most of which get forgotten within hours from posting. We e-publish articles and blog posts, much like this one, that we know will be lost in the noise by tomorrow morning. We build apps that within weeks or days become outdated. There is hardly anything we put together today in the electronic form that is going to get discovered by our descendants a decade from now, never mind a century or a millennium.

The enterprises store years worth of data and process tens of thousands of transactions daily. For many industries, it is no longer possible to go back to using pencil, paper, and handwritten order forms. The financial industry is more reliant on electronics than ever. A major climate event or a World War affecting electronics is bound to disrupt the way we do business as we know it.

I do not know what the solution is. I do know that the humanity created a single point of failure for the entire civilization. Our long term strategical thinking has been reduced to near term instant gratification that will hardly last a generation.

Cloud Power: Operations costs are the Achilles’ heel of NoSQL

Check out the latest post at my Cloud Power blog at Computerworld:

Companies interested in adopting NoSQL should consider their options carefully. The vast majority of database use cases do not need massive horizontal scalability. Most applications could be better off with traditional SQL databases. In the cloud, there are NoSQL alternatives that cost less and are easier to maintain.

via Operations costs are the Achilles’ heel of NoSQL | Computerworld.

Cloud Power: IT departments must transform in the face of the cloud revolution

In case anyone’s been wondering why there hasn’t been an update to this blog in a couple of weeks — well, I am now part of the Computerworld contributor network with a blog called “Cloud Power“.

My first post is on the topic of Shadow IT  :

Cloud computing democratizes developer and end user productivity at the expense of transparency and IT control. Since developers and users are able to provision and utilize resources as needed, it is easy for costs, overall architecture and security to get out of control. Rather than getting in the way of productivity, however, the IT departments must evolve their role from that of the gatekeepers into that of enablers.

Enjoy the read and join the conversation!

Banking Technology is in Dire Need of Standartization and Openness

Old Bank Photo credit Toby Dickens
Old Bank
Photo credit Toby Dickens

A few weeks ago Investors Bank in New Jersey overhauled their systems. As a result Mint became incompatible with Investors and Investors customers could no longer view their account in Mint. There is anecdotal evidence1 that Mint uses the Yodlee platform2 for the integration. As it turns out, there is no standard mechanism by which external applications can work with banks. Yodlee’s own page states:

Through a proprietary system of direct data access and HTML parsing, Yodlee delivers financial data from more than 14,000 sources, and growing.

While the technology world is moving towards open APIs and standard authentication protocols3 the banking industry continues to rely on proprietary systems and HTML screen scraping. It seems that even using Yodlee platform it is not possible to integrate with banks in any standard way. Each time a bank updates their systems a team of engineers at Intuit must update integration scripts to ensure their customers can continue to use Mint with that bank4:

When a financial institution updates their system, our engineers have to rewrite the script on our end to match so that we can continue supporting them. Typically, they are notified when this is going to happen and can get it updated pretty quickly. However, please open a ticket by filling out our Contact Mint form to make sure this is on their radar and they can get the script updated as soon as possible.

The way Mint integrates with banks is by asking users to enter and store their bank credentials. Mint expects us to trust their security5. The technology industry, however, has long established a protocol by which an application (like Mint) needing access to an outside resource (a user’s bank account) does not need to capture user’s credentials. It is called OAuth6.

Had banks implemented OAuth, mint would use the protocol to obtain an authorization from the user to act upon the bank’s API on behalf of the user. In the event of a security breach at Mint it would be possible for the banks to invalidate all tokens — and disable all further access by Mint. Users would gain control over which applications they want to access their data and which they do not.

In 2015 there is no need for HTML screen scraping or proprietary technologies. Would Yodlee platform even be around if the banks used OAuth and standard API7 ? This is an industry that is in dire need of innovation. Banks need to learn how to recruit and retain top talent from the technology companies, not the other way around. They need to look beyond their traditional well accepted consulting vendors and service providers and think outside the box — especially considering the fact that the technology challenges they face have already been solved by others.

I Stand With Ahmed

This week a precocious 14-year old immigrant Ahmed Mohamed wanted to impress his teachers with a clock he made at home. He built it into one of those pencil boxes you buy at a craft store that look like a small brief case. The teachers and school officials thought it looked suspicious and called the police. The police proceeded to arrest him as a terrorism suspect1.

This is a technology blog and so I won’t get into the topics of politics, racism, and terrorism. Let’s even set aside the seemingly incompetent reaction of Irving, TX law enforcement who had not evacuated the school. Instead I am going to focus on the topic of STEM education in the United States.

My 8 year old daughter building an Arduino LCD circuit
My 8 year old daughter building an Arduino LCD circuit

It just so happened that a few days prior to this incident my 8 year old daughter asked if she can bring the Arduino LCD circuit I had built with her to school to show her friends and teachers. I was not even thinking that an elementary school teacher may think a circuit with batteries, wires and a display is a bomb and it may result in her arrest.

To tell the sorry state of American STEM education all one needs to do is take a tour of top engineering universities and visit science and engineering classrooms. A keen observer will find that the majority of students are immigrants. These students have multiple advantages over American students — they come from cultures that value knowledge and education, families that invest in their childrens future, and teachers who can a tell a bomb from a clock.

Of course, what starts in universities transfers to workplaces. A visit to any software company or even an IT department just about anywhere will reveal that the majority of developers are immigrants as well. They come from India, China, Ukraine, Belarus, Russia, and elsewhere in Asia and Europe.

Meanwhile, American politicians draw crowds of people at campaign rallies fanning the flames of fear over American jobs2. The reality, however, is that a much bigger threat to the future of American middle class jobs starts in schools. When teachers, school, and law enforcement officials can’t tell the difference between an explosive and a homemade clock — how can American kids look up to them ?

Setting Up Cross-Region Replication of AWS RDS for PostgreSQL

As of today AWS RDS for PostgreSQL1 does not offer cross-region replication. Short of switching to one of the RDS offerings that do support it, there is a few options to consider.

1. Custom Configured EC2 Instances with Master-Slave Replication

Master/slave light rail
Master/slave light rail

This setup sacrifices the benefits of AWS RDS service in exchange for greater control over replication settings. In this setup, one region hosts a master PostgreSQL host, and another region hosts a slave which can also act as a read-replica2.

Advantages

Greater control over replication settings.

Disadvantages

  • Give up all the advantages of running in AWS RDS environment.
  • Writes can only be performed in the master region.

2. Software-defined Two-phase Commit

Commitment Photo credit: Ed Schipul
Commitment
Photo credit: Ed Schipul

In this setup there are two independent AWS RDS instances. The application, however, utilizes a two-phase commit protocol3 to guarantee that all writes make it into both databases in a transactional fashion.

Advantages

  • Simple configuration
  • Does not sacrifice any of the AWS RDS advantages

Disadvantages

  • Responsibility for ensuring that writes make it into all regions fall onto the application itself.
  • Increased application code complexity.
  • Write performance is sacrificed since all regional databases must participate synchronously.

3. Asynchronous Writers

Fanning Out Photo credit: Tim Haynes
Fanning Out
Photo credit: Tim Haynes

In this approach each region hosts an asynchronous writer that listens on an SQS queue4. All writes are published on the SNS topic that is configured with all regional writer queues as subscriptions5. When the application running in any of the regions wants to write into the database it publishes a message on this SNS topic which then fans it out to all of the regional SQS queues.

Advantages

  • Simple configuration
  • Does not sacrifice any of the AWS RDS advantages
  • Does not sacrifice write performance

Disadvantages

  • Subject to software bugs
  • Subject to SNS and SQS bugs and outages
  • No guarantee of consistency
  • Requires a mechanism for periodically reconciling differences between regions