Dependency Upgrades: Why Software Engineers Need A Policy And Process

For most software engineers, dependency management includes upgrades and best practice suggests upgrading dependencies proactively. In order to ensure a smooth process and one that is maintained properly, it is best to have a well-defined policy and process.  

Why you need a policy and process  

All software depends on other software to accomplish useful tasks. Even the smallest  hello world  program depends on the libraries of the language it is written in. Typically, as a piece of software  gets more sophisticated it depends on more third-party software or libraries. Leveraging existing third-party libraries is a standard practice for engineers  to enable faster development.    

However, vulnerabilities and bugs of dependencies become vulnerabilities and bugs of the software the dependencies are included in. As a result, dependencies must be chosen with due consideration, and constantly monitored.  Best practice is to not only upgrade dependencies proactively, but to also be on the latest versions of libraries as often as possible.

Why Upgrade Dependencies?  

The single biggest reason for upgrading dependencies is  security. Most software has vulnerabilities that are gradually fixed over time. Upgrading dependencies, is  the  simplest way  to avoid security issues. The latest versions can introduce new vulnerabilities that are not yet discovered. However, it is still safer to upgrade in order to fix known vulnerabilities. 

Most of the  changes between versions  are  bug fixes  and  performance improvements, that’s another good reason to upgrade. In recent tests, a docker version upgrade gave a significant boost to the performance numbers.  

Other reasons to upgrade include being able make use of new features, avoiding giant upgrade steps as it is always easier to update in smaller increments, and it is also easier to upgrade a single dependency if everything else is up to date.  

Upgrade Experiences

Software developers should be upgrading dependencies on a regular basis. Here are a few examples of good, bad and ugly dependency upgrades  that inform the upgrade process. 

The Good  

Overall library upgrades  should be  smooth or have relatively few issues. Good libraries maintain backward compatibility, and provide suitable replacements for any APIs that are deprecated. As a result, upgrading such libraries should  involve  only  minor code changes. Good migration guides, and well-crafted deprecation warnings  make this process relatively painless. Recent examples of these include various Apache Commons libraries,  Mockito,  and  RxJava.   

The Bad   

Every now and then an upgrade does not go according to plan, and unexpected issues arise due to not having the bugs or changes well documented.   Recent example of this occurred with an upgrade to Guava version 25.  Guava upgrades are typically seamless and  jumping to version 25 was expected to be a simple version bump.  

However, intermittent errors occurred in performance tests because of a misbehaving cache. The intermittent nature of the errors meant that they were difficult to track down, resulting in a days long testing to locate the issue. To allow enough time to deal with such surprises, it is best to do upgrades at the beginning of a release cycle.

The Ugly  

Sometimes  an upgrade can get ugly.  This is typically the case when a number of dependencies have to be upgraded in lockstep. This problem will be  exacerbated  if regular upgrades are not done, and several upgrades of versions need to take place in one go.  

As an example, a recent Spring upgrade required upgrading Cucumber. Upgrading Cucumber required upgrading Spring. Also resulting in needing to upgrade Camel, Junit,  Servicemix, and Gemini. Attempting to untangle this dependency web was challenging.    

Automation  

Upgrading dependencies is not the most exciting work for an engineer. To ensure upgrades are quick and happen often, it is strongly recommended to leverage all automated tools available.  

For example, Versions Maven Plugin is best to use for maven projects  to generate reports listing the libraries that have new versions available. The plugin can also auto-upgrade the versions directly in source code. Most languages and frameworks have similar tooling available.  

It is worth noting to only use the auto-upgrade feature to do revision, and minor version upgrades. These upgrades are typically quick and require minimal code changes. Overall, the plugin is a huge time saver.  

A major version upgrade is often a more conscious decision to upgrade with the understanding that such an upgrade will require some work.  

New vulnerabilities are discovered  every day, and it is hard to keep up with without some  automation.   

Use a security and  license  compliance  tool, e.g. Black Duck, to  identify vulnerabilities and license compliance issues in the open source libraries.   

Lastly, it is recommended to create a robust suite of functional and performance tests that ensure library upgrades do not have negative consequences.  

The Upgrade Process – what a strong process looks like  

1. Identify the libraries to be upgraded.  Leverage an automated tool in this step. Most languages and frameworks have one. E.g. Use Versions Maven Plugin  for Maven projects.   
2. Ensure version being upgraded to does not have known security vulnerabilities, and license issues. Before changing a single line of code one can verify candidate versions using a security compliance tool  e.g.  Black Duck.  
3. Do the bare minimum code change to upgrade the version i.e.  fix  any compilation errors, and any  functional  test failures. Don’t try to use any new features at this point.  
4. Verify performance. Run your whole performance test suite.   
5. As much as possible, do the upgrades at the beginning of  a  release  cycle  to give  you  time to live with any surprises that might come up.   
6. As an optional exercise – use the new features. Take a look at release notes of the new versions, check if there is anything that catches the eye and would be helpful. Use your judgement to determine if it might be useful to spend a bit of time on this.  

In  closing, be sure to continue to learn from experiences and fine tune this process.

Note: This was originally posted on GetElastic - Enterprise Commerce Blog.

Testing for Exceptions in RxJava 2.x

RxJava 2.x provides useful assertions to test for exception conditions. Assume we have the following class with three methods that result in three different error conditions

1. No Error
2. An exception
3. An exception with a distinct property, in this case an exception message.

We can write one test for each of the three error conditions by using the different asserts available in RxJava

1. assertNoErrors() 
2. assertError(Class<? extends Throwable> errorClass)
3. assertError(Throwable error)
4. assertError(Predicate<Throwable> errorPredicate)

The ability to write custom errorPredicate is extremely useful when the same exception type is thrown from several locations but each exception instance can have a different property such as the exception message.

Tip for writing RxJava code for beginners

Let's say you are using RxJava (or RxAndroid) and your goal is to write code which looks something like the foo method shown in the snippet below. Once you are experienced in writing functional code this appears trivial, however for programmers used to imperative programming style this can appear daunting.

Here is a step by step process which should hopefully make things simpler.

Step 1: Invoke the first method in the desired chain, and use the IDE to assign it's return value to a local variable. This way you can see the type of this expression.

Step 2: Use the local variable created in Step 1 to invoke the second method. If you are unsure about map vs flatMap, simply use map by default. Again, using the IDE assign the result to a local variable. In this new variable a Maybe is nested inside an Observable.

Step 3: This nesting is a hint to change map to a flatMap operation.

Step 4: Follow the above steps for the remaining method calls in the goal.

Step 5: Simply use the IDE to inline all the local variables to reach the goal.

RxJava: Converting to Single type

In RxJava Single is a special type. It must emit exactly one value. Other types like Observable and Maybe can emit no values. If one is not careful these properties can lead to runtime errors in the code.

Converting a Maybe to Single

As shown by the tests in these examples this will rightly result in error.

Converting an Observable to Single

Oddly enough as converting an empty Observable to Single does not result in an error. Maybe that's a bug in RxJava implementation? A quick read of the code does seem to indicate that.

However, the javadoc of Observable#flatMapSingle says "Maps each element of the upstream Observable into SingleSources", and the method returns an Observable. So the behaviour is correct, the second snippet with a non-empty Observable shows that. Maybe the methods could have been named better?

Converting an Observable which emits too many values to a Single rightly results in an error.

Converting a Flowable to Single

Flowable exhibits same behaviour as Observable as shown in these snippets.

Converting a Completable to Single

Just for completeness (no pun intended), this works as expected.

AngularJS and Amazon S3 & CloudFront - HTML5 mode and Social Sharing

The Setup
You have a single page AngularJS application hosted on Amazon S3 and CloudFront. You have used one of the two possible configurations, i.e. with or without static website hosting enabled on S3.

Problem
Sharing a link from your website on Facebook, Twitter, Linkedin etc does not work well, because none of them run JavaScript to understand a shared link. This is a huge problem because in a Single Page Application (SPA) you set the page titles and descriptions using JavaScript.

A minor problem is also that your website urls are not pretty and include hashbang symbols i.e. # or #!. Besides not looking nice, URLs with hashbang symbols also do not work well when links are shared on any of the social networks.

Solution
The first part of the solution requires enabling HTML5 mode in the AngularJS application. Once HTML5 mode is enabled your links will change from

http://example.com/#/path   TO  http://example.com/path

Client Side Changes
This part is simple as shown in this gist


Next make sure all the links in your app are correct
1. Change # to / in all hrefs in your app. This should be easily done via find and replace. Find matches for 'href="#' and replace with 'href="/'
2. Make sure any assets included in the apps' index.html have a href starting with /. That is change any href values from "path/to/css/ to "/path/to/css"

Server Side Changes
Now when you open the home page and use the website by clicking around everything should work fine. However, if you copy paste a link directly in the browser you will get a 404. To fix this 404 error we need some server side changes.

 - Development (Grunt)
Quite likely you are using Grunt on your dev machine. First install connect-modrewrite and then add a middleware to the livereload section as shown in this gist.


 - Production (Amazon S3 and CloudFront)
Assuming you do NOT have static website hosting enabled on S3. All you have to do is create couple of Custom Error Responses on CloudFront.



Detect JavaScript Support
We filter out the social networks by providing them with a static HTML page containing all the relevant meta tags, and also a simple JavaScript snippet that redirects JavaScript-capable browsers to the root URL, with the correct route hashed out, so that CloudFront & S3 take you to the application and Angular interprets the route correctly. An example of such a file is shown in this gist.

Now when you share the link "https://example.com/path/index.html" social networks will be able to create nice cards. When a user clicks on this link, they will be redirected to the right place.

Notice that the shared link includes "index.html", we can get rid of that if we enable Static Website Hosting on S3. More on that later...

Refrences
1. https://medium.com/@gigbloc/deep-linking-rich-social-sharing-seo-with-angularjs-and-amazon-s3-82f99cf83cb5#.fl6miptq1
2. http://www.ericluwj.com/2015/11/17/seo-for-angularjs-on-s3.html
3. http://www.ericluwj.com/2015/11/10/angularjs-with-html5-mode-on-s3.html
4. http://stackoverflow.com/questions/24283653/angularjs-html5mode-using-grunt-connect-grunt-0-4-5

Deploy a static web app to AWS Cloudfront and S3

There are two ways to deploy a static web apps to AWS CloudFront and S3.

1. Enable Static Website Hosting on S3. Once this is enabled, you can also setup CloudFront to point to S3 website endpoint. Make sure this is not the standard S3 endpoint. Here, CloudFront is simply caching the static files. All files in S3 will also be accessible without CloudFront.

2. Use CloudFront to point to the standard S3 endpoint. This way you make S3 bucket contents available only to CloudFront. In this scenario CloudFront both caches the files and is also a gatekeeper to all the files.

What to use for login - username, email or phone number?

It seems trivial on surface, pick a strategy and go with it. However, it can be useful to pause a think about your exact use case as each of username, email and phone number have distinct pros and cons.

If you are developing a mobile app, phone number along with a one time password is a good option to use for login. Whatsapp, Telegram and others use this strategy. The clear advantage is that most users have a phone number (people with tablets may not have a phone number), and a user does not have to remember a password. However, phone numbers can change often, especially if a user moves to a different city or country. As a result, a user can lose access to their account unless the app allows to change phone numbers by some means. Moreover, a user's phone number may have previously belonged to someone else. This may result in a user getting access to someone else's account. Users may need to provide an email or an alternate phone number to help avoid these pitfalls.

Username along with a password is a good option both for mobile apps and web apps. A user can choose a unique username and a password. Now no user can access another user's account by accident as in the case of phone numbers, and the user does not lose access to their account as long as they remember the username and password. However, remembering usernames and passwords for all the different apps is hard. An email address or phone number would be required for username and password recovery. One big advantage of usernames is that users can try your app without providing their personal information i.e. email or phone number. Also, the registration step can be really quick, if users don't have to verify their email or phone number to start using the app or wait for a one time password. Snapchat uses this strategy. At the time of registration it asks for username, password and email, but does not force you to verify the email before using the app. You can verify your email later (you may not have email setup on your phone) so that you can recover your account details in case you forget them.

For web apps, email along with a password is a good option. Emails should typically change less frequently than phone numbers. (Some people use the email from their ISPs and hence end up changing it frequently, but I suspect they are in a minority). If emails are used, users don't need to remember a username and an email is already unique, so a user does not have to spend time coming up with something unique as in the case of usernames. Recovery of account details with an email is also simple. However, it is a good idea to get a user to verify their email address. Some apps force users to complete this verification before giving them access to the app, which lengthens the registration process. On the other hand, if emails are not verified, you may end up in a situation where another user has created an account with your email address, and hence you cannot use that email address to create an account. Users on popular apps like Facebook often face this situation. Email with a password can also be used for mobile apps, however it is useful to remember that a user may not have email setup on their phone hence may not be able to verify their email right away, or access an one time password.