Big Data Should Be Used To Make Ads More Relevant

It is 2015 and by now advertising agencies could have figured out the best way to deliver relevant ads to viewers. They have not. Hulu is the prime illustration of this phenomenon.

We canceled our DirecTV subscription and declined to use Comcast for anything more than Internet access for over three years now. When I watch shows on Hulu I get ads that I never act on because they are so irrelevant.

The most irrelevant ad on Hulu is the one advertising Hulu Plus itself. I am already a subscriber and they should be able to tell. The second most irrelevant ad is the one telling me, repeatedly, that they got their hands on the full collection of “Seinfeld” episodes – which I am already four seasons into!

While watching Hulu it is also not uncommon for me to get advertisements for shows that are only available to cable subscribers. But I am not, and I made it clear that there is no circumstance under which we will subscribe to cable television. Shows not available on Netflix, Hulu or Amazon Prime might as well not exist. There is not going to be a conversion in response to Xfinity TV ads on Hulu from yours truly.

Hulu product advertisements leave much to be desired as well. Why advertise Microsoft products to someone who uses Apple products and Linux ?

There is so much Hulu could do to create more relevant ads for me. Hulu knows well enough who I am and what my interests are. They know my IP, they can search the Internet for my name. They can give me a way to disable irrelevant or annoying commercials.

The age of Big Data and voluntary surrender of privacy should have brought more meaningful and more relevant advertising. The knowledge the advertisers have about consumers is unprecedented. They should use it.

Social Media Detox

I am going on a two week vacation to celebrate our 10th wedding anniversary. One of the things I would like to accomplish is a clean up of my social media use. Another is a detox of sorts: I want to see if I can give up daily checking and participation in Facebook.

I use my personal page on Facebook for communication with close friends and family. I do not feel comfortable having coworkers on it. I keep my Personal Facebook page mostly private. I do not want a situation where things I do at work to spill into my private life, and things I do in my private life to spill into work. I don’t want to constantly wonder if something that I say to my close friends and family may touch a nerve with someone at work.

For professional networking I use Twitter, LinkedIn, github, WordPress, StackOverflow, and a professional page on Facebook. I am extremely careful with what I say in these forums and I make sure my posts are based on publicly available information. Nothing I say in that media should be construed as an opinion of or about any of my employers, past, present or future1. I voice opinions about technology, science and processes. I do not voice opinions about individuals.

With all that said, please – if you in the past followed me through my personal Facebook page and found yourself unfriended, don’t take it as an offense. It means I respect you as my professional contact and I wish for our relationship to stay professional. Please follow me using professional networking tools.

The Three Myths About JavaScript Simplicity

Jersey City -20100723-21.jpgThere is a perception among many in the software industry that JavaScript is simpler to learn and use than, say, Java. I’ve even heard some say that JavaScript developers are easier to recruit.

While there are many myths about JavaScript that detractors cite as reasons not to use it, there are also at least as many myths propagated by advocates of this technology. These are the most common ones I’ve heard over the years.

1) JavaScript Developers are Easier to Recruit

That only holds true if you are trying to hire a 17 year old who built simple websites out of his mother’s basement. Yes, he knows JavaScript, and yes he’ll take you up on that low ball offer that no experienced developer will accept. Will you get a timely professional result ?

Lisa Eadicicco writing for Business Insider shows that average JavaScript developer salary is right up there along with C++ and Java:

  1. JavaScript – $91,461
  2. C++ – $93,502
  3. JAVA – $94,908

A cursory search of LinkedIn reveals almost 5 million people listing Java as their skill, and less than 3 million who list JavaScript as their skill. By this statistics alone, one is almost twice as likely to recruit a Java developer than they are a JavaScript developer.

2) JavaScript is Easy to Learn

By the law of Turing Equivalency most programming languages are equivalent and if you know one you can learn any other. The complexity is never in the language itself – it is in the frameworks and libraries.

I am writing this post in 2015 and JavaScript has been powering web apps for at least 20 years. One would think that by now handling HTTP and building MVC apps would be part of the platform. Yet, JavaScript leaves much to be desired.

JavaScript, for example, has multiple libraries for HTTP REST requests. In Node.js it is not uncommon to use one library on the server and a totally different one on the client. Consider the multitude of single-page app MVC frameworks – each one has a drastically different philosophy of using it.

The difference between null and undefined and between == and === as well as lack of type safety leave much to be desired. The developer has to constantly keep those nuances in mind, as if they have nothing better to do.

3) Non-Developers Can Learn to Use JavaScript

This argument usually goes along with myth #2. The only advantage JavaScript has over other languages is that the most one needs to get started writing in it is a text editor and a web browser. That is not something to be overlooked — JavaScript really is an easy language to get started in because of that.

More often than not developers will learn the business domain of their apps long before business users will learn how to program. There are languages that business users may be more comfortable writing code in — SQL comes to mind, for instance, and perhaps the data and reporting APIs can and should be built by the business analysts.

So Why JavaScript ?

The points above should not be a reason not to use JavaScript. Reality is that when it comes to building dynamic webapps and microservices there is no choice other than to use it. It is a great tool for rapid prototyping and for building backend services using platforms like AWS Lambda. Despite what I said, it is a great language and has many useful applications — but make no mistake, it is a programming language like any other.

Book Review: “Shop Class As Soulcraft” By Matthew B. Crawford

Matthew B. Crawford's Shop Class as Soulcraft: An Inquiry Into the Value of Work showed up in my recommended books list on Amazon and it looked intriguing. I've read it, cover to cover, and it turned out to be one of the most influential books of my career.

I didn't go into software engineering because it is a dynamic field or because I enjoy working with eccentric characters. I am a software engineer because all my life the one thing that makes me happy when it comes to work is building something with my hands.

As a kid my favorite toys were erector sets. As I've grown they got replaced with short wave radio kits and finally in my early teens by a computer. Even though “Shop Class as Soul Craft” is about manual work I find it applicable to software engineering. Maintaining a motorcycle requires very similar cognitive skills as maintaining, say, an order management system software.

Crawford questions the value of higher education. We are indoctrinated from early on that we must become knowledge workers. We go to college. We get graduate degrees. Now, consider thousands of MBAs getting entry level administrative jobs that pay far less than, say, entry level computer programmer jobs and you get the point he is getting at.

Why, Crawford poses a question, does a plumber charge you $80/hour while a finance major would be lucky to start at $30/hour ? The difference is that a plumber does something tangible.

In software engineering there are hundreds of books written on the subject of division of labor. “JEE 6 Tutorial” starts off with the description of technologist roles each one dedicated to their compartmentalised effort1. Well, consider this:

Once the cognitive aspects of the job are located in a separate management class, or better yet in a process that, once designed, requires no ongoing judgment or deliberation, skilled workers can be replaced with unskilled workers at a lower rate of pay.

But while the direction of cognitive thought is moving away from the so called knowledge worker, the management is expending a lot of energy on avoiding responsibility for the decision-making aspects of their work:

…one of the principles of contemporary management is to “push details down and pull credit up.” That is, avoid making decisions, because they could damage your career, but then spin cover stories after the fact that interpret positive outcomes to your credit. To this end, upper management deals only with abstractions, not operational details. If things go well: “Finding cross-marketing synergies in the telecommunications and consumer electronics divisions has improved our strategic outlook heading into the fourth quarter.” If things go badly: “Change the Vonage display? That was the kid’s idea. What’s his name, Bapu or something. Jeezus, these immigrants.”

As software engineers many of us consider working on brand new projects a more glorious and satisfying career. Here is the catch – brand new projects require far less cognitive ability and can be labor-divided and outsourced. Maintenance projects, on the other hand, require constant dealing with something not of our own making – much like car mechanics or plumbers jobs. Crawford explains:

Fixing things, whether cars or human bodies, is very different from building things from scratch. The mechanic and the doctor deal with failure every day, even if they are expert, whereas the builder does not. […]

Because the stochastic arts diagnose and fix things that are variable, complex, and not of our own making, and therefore not fully knowable, they require a certain disposition toward the thing you are trying to fix. This disposition is at once cognitive and moral. Getting it right demands that you be attentive in the way of a conversation rather than assertive in the way of a demonstration .7 I believe the mechanical arts have a special significance for our time because they cultivate not creativity, but the less glamorous virtue of attentiveness.[…]

Any discipline that deals with an authoritative, independent reality requires honesty and humility. I believe this is especially so of the stochastic arts that fix things, such as doctoring and wrenching, in which we are not the makers of the things we tend. Similarly, in art that is representational, the artist holds herself responsible to something not of her making. If we fail to respond appropriately to these authoritative realities, we remain [ed: absent minded] idiots.

A prime example from my personal career is learning compiler construction. In college we were taught how to build our own compiler from scratch using lex and yacc. I learned a lot more about compiler construction in graduate school where I had to modify an existing Ada compiler. Likewise, I find that software engineers who always built enterprise systems from scratch and never had to support and maintain an existing one are hardly qualified to do their jobs.

Crawford points out:

We have all had the experience of dealing with a service provider who seems to have been reduced to a script-reading automaton. We have also heard the complaints of employers about not being able to find conscientious workers. Are these two facts perhaps related? There seems to be a vicious circle in which degraded work plays a pedagogical role, forming workers into material that is ill suited for anything but the overdetermined world of careless labor.

Management can't expect workers to be conscientious when their work is reduced to following instructions, processes and procedures. Software engineer can't be conscientious when their work is separated from interaction with business users, sponsors, and analysts. In other words, a good engineer has to eat their own dog food. On the issue of teamwork and office culture, Crawford explains:

we are accustomed to think of the business world as ruled by an amoral bottom-line mentality, but in fact it is impossible to make sense of the office without noticing that it has become a place of moral education, where souls are formed and a particular ideal of what it means to be a good person is urged upon us […] Where there isn’t anything material being produced, objective standards for job performance are hard to come by. What is a manager to do? He is encouraged to direct his attention to the states of minds of workers, and become a sort of therapist. […] one finds an imperative for the manager to care, and to sincerely hold forth to his subordinates the possibility of personal transformation. He is not so much a boss as a life coach. The contemporary office requires the development of a self that is ready for teamwork, rooted in shared habits of flexibility rather than strong individual character. […]

Workers must identify with the corporate culture, and exhibit a high level of “buy-in” to “the mission.” The division between private life and work life is eroded, and accordingly the whole person is at issue in job performance evaluations.

On career vulnerability of managers:

Managers are placed in the middle of an enduring social conflict that once gave rise to street riots but is mostly silent in our times: the antagonism between labor and capital. In this position they are subject to unique hazards. […] Rather, [ed: their] authority is embodied in the persons with whom one has working relationships up and down the hierarchy […]

One’s career depends entirely on these personal relationships, in part because the criteria of evaluation are ambiguous. As a result, managers have to spend a good part of the day “managing what other people think of them.” With a sense of being on probation that never ends, managers feel “constantly vulnerable and anxious, acutely aware of the likelihood at any time of an organizational upheaval which could overturn their plans and possibly damage their careers fatally,” as Craig Calhoun writes in his review of Jackall’s book.9 It is a “prospect of more or less arbitrary disaster.” […]

A good part of the job, then, consists of “a constant interpretation and reinterpretation of events that constructs a reality in which it is difficult to pin blame on anyone, especially oneself,” according to Calhoun. This gives rise to the art of talking in circles. Mutually contradictory statements are made to cohere by sheer forcefulness of presentation, allowing a manager to “stake out a position on every side of an issue. Or one buries what one wants done in a string of vaguely related descriptive sentences that demand textual exegesis.” […]

When a manager’s success is predicated on the manipulation of language, for the sake of avoiding responsibility, reward and blame come untethered from good faith effort. He may then come to think that those beneath him in the food chain also can’t be held responsible in any but arbitrary ways.

One might be tempted to think this is demoralizing for all involved. But we are highly adaptive creatures, and these circumstances generate their own sort of morality, one in which the fixed points of an internal moral compass must give way to a certain sensitivity and nimbleness. Managers may continue to have strong convictions, but they are obliged to check them at the door, and expect others to do the same. “[M]oral viewpoints threaten others within an organization by making claims on them that might impede their ability to read the drift of social situations.” As a result there is social pressure (one might say a moral demand) not to be too “moralistic.” This pressure is rooted in the insecurity of managerial careers.

Crawford describes his experience of getting paid for motorcycle repair – where a wad of $1000 in cash given to him by a happy client was somehow more rewarding than the same amount given for an office job. The reason is because the motorcycle either runs or it doesn't, while at the office he was not being judged based on the results of his work.

First time I got paid for building a piece of software was when I was 13 and I wrote a Lotus 1-2-3 macro for a small business and got paid $50 for it. When I worked for a couple of years as a consultant getting my timesheet signed every week and realising that my customer will be utilising my services for another quarter was highly rewarding. Likewise, I am uncomfortable in environments where metrics for performance and compensation are arbitrary and subjective, as most engineers should be.

Whether you agree with Crawford's views on corporate culture or not he does offer a sobering and honest viewpoint on the modern work ethic. It is a must read for anyone considering themselves an engineer and even a manager. It is one of those rare books that offer a society a look in a mirror undistorted by modern culture.

  1. Note how Java EE 7 tutorial does not include a section on development roles, which is a sign of evolution of management techniques in recent years 

Attracting STEM Graduates to Traditional Enterprise IT

Techcrunch reports:

If 17-year-old Google is at all worried that it’s losing its mojo, it should find some new data highly reassuring. According to a survey of 19,000 students across 340 universities around the world, Google is still their top choice when asked where they’d want to work.

Their other top picks, in descending order: Apple, Microsoft, Facebook and Amazon.

Apple, Microsoft, Facebook and Amazon are not the only ones in need of top Computer Science talent. Financial companies have plenty of openings. Just about any medium-large sized traditional enterprises advertises openings for top talent as well. However, as the Techcrunch article reports, no Computer Science student wakes up every morning saying “I dream of working for a bank on nightly batch jobs!” or “I dream of working for a paper products wholesaler configuring ERP systems!” — this is just sad reality. If every CS major only wants to work for the top 5% companies, that leaves the vast majority (95%) of the companies struggling to attract and retain the top talent.

A person saying “I am enthusiastic about building software products” is very different from a person saying “I am enthusiastic about solving business problems using software products.” Former wishes to build tools, the latter wishes to solve problems with tools. There is a reason why some schools have a degree program called “Information Systems.” Scanning through the Rider Univeresity Information Systems program we see topics like this:

  • Apply practical knowledge in their use of computer-based productivity tools.
  • Demonstrate an understanding of enterprise integration applications such as SAP R/3.
  • Explain the value of electronically integrating the major functional areas of an organization in order to facilitate more effective management decision-making.
  • Rider’s Department of Information Systems & Supply Chain Management is a member of the Oracle Academic Initiative, the SAP University Alliance and the Microsoft Developer’s Network (MSDN).

Rather than being a Computer Science program, this is a program in management and information systems — and that is what the majority of the traditional enterprises demand. In fact, trying to recruit a Computer Science major for such openings would be doing a disservice to both the employer and the candidate. The employer would find it difficult to staff traditional enteprise IT openings with Computer Science graduates who will in turn find it less desirable to work on such projects.

The needs of enterprises don’t stop at IT, however. Custom internal applications and algorithms can be a major differentiator among competitors. In sales ability to anticipate and predict customer product demand can be the difference between a closed sale and a missed sale. Ability to optimize truck routes can mean millions of dollars in savings. In finance market analytics can make a difference between a successful investment and a failed one. So how does a traditional enterprise attract STEM graduates who wouldn’t normally look for a job there ?

Oleg’s Utilities : Exploring Node, Angular and Heroku

I needed a sandbox where I could experiment with Heroku, Node, JavaScript, and Angular and so I created a little suite of apps that I call “Oleg’s Utilities“. For now it is just a date calculator that my wife told me would be useful to her, but I’ll add more. A few observations:

  1. I am not an experienced web developer, but I went through Angular and Node tutorial and was able to get started with a simple app in about 20-30 mins.
  2. First time ever using Heroku, I was able to deploy a useful app in minutes.

It is amazing what one could do these days with the cloud services. As I pointed out before it truly is a Kodak Moment for IT departments and technology vendors. The fact that anybody can put together an app and deploy it to the cloud in minutes, and essentially zero cost to them is a truly disruptive revolutionary technology.

Your IT Department’s Kodak Moment

Your IT department’s Kodak moment is now, but it is not the kind of a moment where you get to take a cute picture and save it forever.

George Eastman founded Kodak in 1888. The company was dominant during most of 20th century in the market for photographic film. Even though they invented the first digital camera in 1975 they dismissed the idea of digital photography. As a dominant player in the industry they did not want to introduce anything that would threaten their near-monopoly on film products. While consumer electronics companies with no vested interest in film introduced amazing digital cameras, Kodak fell into a pattern of steep decline in the late 1990s and in 2007 had to file for bankruptcy.

Today’s enterprise IT market is monopolized by on-premise data centers. It is dominated by big vendors that have vested interested in maintaining the status quo. They would all love to tell you that they have some sort of a magic solution that brings the cloud to you. Complacent enterprise IT departments are more than willing to listen – after all, IT view themselves as gatekeepers to technology adoption in their companies.

The reality is that they will never keep up. The cloud brought the barriers to entry to near zero. While it used to be that it would take months or years and millions of dollars for a company to scale out their on-premise IT, now the same takes hours or days and zero upfront costs to scale out a data center. Companies that adopt cloud services will find themselves delivering applications, tools, and products to their customers much faster and at a lower cost. Companies that continue to look for excuses not to will find themselves outcompeted by peers that do not.

This is not limited to software technology companies, although they will feel the impact first. IT departments at companies to whom software is more of a tool than a product are at danger of rendering themselves obsolete by resisting cloud adoption. For a business unit to build and deploy an application IT is no longer required — all they need is a budget and an internet connection. IT departments, therefore, could make themselves more useful by facilitating API and data integration with cloud applications rather than standing in the way of progress.