Hi 👋 I'm Joakim


https://antman.dev

Dependencies, what you think?

Dependencies - an asset and a curse

a story about losing hope to regaining control.

The early days

based on git history and stories from the elders

The ship is leaking

let's abandon build another ship

The cracks are still there

and still leaking

image

Hope is really lost

No new features should be added to the old codebase

Rising as a Phoenix, but probably slower

and for sure not as elegant

One ship and tons of boats

and some are sinking

We care as a team

Not shared responsibility, but the responsibility taken

Visibility, a goal and a promise

Aim for 🟢 ,react on 🟡 and cringe on 🔴

We promise that

All apps running on the latest patch version of a supported Ruby version with a libyear age of max 20 years with no open and fixable security issues.

Did we learn anything

Dependencies are important but risky

Losing hope is not a reason, it's a feeling

Making promises and goals visual and visible motivates people

Im Joakim or Jokke Nice to meet everyone and thanks to the organizers taking the time putting these kind of events together. Was thinking introducing myself first a bit, from a techy angle. Was introduced to web development via Perl, at that time it was probably called building homepages and I cannot remember really what I was doing but probably had to do with collecting data from form. Started my paid career back in C/C++ in the Microsoft ecosystem making Windows UI apps. Got dragged into the Web again because of .NET and C# When this happened, OSS dependencies and OSS in general became a thing, At that time I my first OSS contribution. Not sure if I dare to say what it was, a jQuery multiselect plugin BTW jQuery 4.0.0 is coming! It was quite a change moving into the Ruby world when. This was when Rails 4.2 was a thing. Nowadays i speak Ruby and im a semi-active maintainer of the ruby-jwt gem.

https://blog.code-cop.org/2014/10/visualising-maven-dependencies.html

Im going to talk about one form of dependencies, but lets try to throw in some interaction at this point and I would like to hear what you think when you hear the word dependency? Food, humans need energy. Im dependent on the first cup of coffee in the morning When it comes to Software dependencies they often enable us to focus on the problem at hand instead of solving small or big technicalities. Like xml parsing or how the HTTP protocol works. Today we going to focus on the latter and not the how dependent we are on the first cup of coffee in the morning to be productive.

https://pixabay.com/vectors/boat-galleon-ship-marine-maritime-2858982/

The main character in this story is a service on the Internet with some APIs and UIs. Enabling businesses to exchange transactions between each other in a efficient and secure way. What it does is not really important, could be any service out there. There will be a bit of Ruby and a bit of Rails, and the pain that a team has felt to keep up to speed with the world and not keeping dependencies fresh. Some battles lost and some won. Some wounds still healing. Lets say that it's based on a true story, so not everything is 100% fact checked.

https://pixabay.com/vectors/people-cooking-fire-flames-36558/

Built on Ruby 1.8 and Rails 2.0 by a relatively small team. With that combo it gained momentum in the business for years to come. Features piled up on feature from success to success growing to serve millions of requests per day. And this talk being about dependencies here we can mention that even one Rails upgrade was squeezed in, the update to Rails 2.3 happened a few months before Rails 4 was released and Rails 2.3 got it's last patch release and was officially end-of-life. One day the service and team was bought by a big corporation that probably was in the impression that they got the most high-tech solution out there.

https://pixabay.com/vectors/water-tap-water-silhouette-drops-1066629/

Now with a new home an new expectations. A whole lot of new pressure. Stakeholders are asking for new fancy features. Team size grows, new people are introduced to the service. Developer persons have heard or even used of new and shiny APIs for ActiveRecord. BUT service running still on Ruby 1.8.7 and Rails 2.3, new tech features were not that easy to introduce and new developers had to adjust using older APIs. Ruby 1.8.7 was EOL already but it is "impossible" to upgrade because of encoding reasons, people have tried and died trying. Impossible things have a hard competition to beat the requests of new and valuable features. This was at the time the hype of microservices was peaking and it was probably easy to convince everyone that lets just start building these new things into separate new and shiny apps so we can use the latest Ruby versions and new and fresh functionality. Let's not go into more details about the decisions during this time. But lets just say that as a result of this era there was a fleet of different type and sized services in our pond.

https://pixabay.com/vectors/crack-lines-texture-angle-lawn-6769438/

Even with these new services servicing the purpose of fulfilling new requirements we still have the reason why we are getting our salaries. The original boat is the one that keeps us a float. So bugs need to be fixed features added to existing older functionality. People still need to run test with the old Ruby versions. Not to mention that we still need to run the thing in production. Daily developer activities as searching for solutions on the Internet was not trivial anymore finding something but soon to realize that the version we are running does not have that functionality. Or a problem has been solved in a more later version of a dependency that is out of our reach because of old runtime versions. One recurring issue was SSL/TLS on 1.8.7, it was a ticking timebomb. Remember once when a integration to a third party broke because an SSL certificate update. Think the issue was SNI extension (Server Name Indication). Luckily we somehow figured out they were running on Azure and we were able to use their MS provided host and certificate to access the service. Many separate apps emerged just because it was impossible to interact with the service from Ruby 1.8.7. Imagine all the OSX updates, trying to cram in a Ruby 1.8.7 build combined with Rails 2.3 and god knows what other 10 year old dependencies were in the mix. Found this little gem of notes from our instructions.

https://pixabay.com/vectors/pray-plead-call-upon-for-help-answer-2669772/

OFC this was not the only battle we were fighting as a popular and growing service. We had a list titled "Scale or die" that listed all of of our main points to ensure scaling for the future. The list had all kind of themes, like load balancer needing renewing, put everything in containers, Database is too small etc. Even one said "more microservices", whatever was meant with that :) But the indication of giving up was really the entry "Stop adding features to the boat" - "Get rid of old Ruby codebase". We had officially lost hope.

https://pixabay.com/vectors/bird-phoenix-fire-flames-feathers-6558500/

Rewrites never work, at least usually. Think we kinda realized that and the "no new features" was just empty words in a document. So even if there were lost hope we still continued to grind with the old Rails app. It was the home to our core functionality, like account handling. Still always being there with a glimpse of hope was the Rails LTS project keeping the engine running on modern Ruby versions. So we could be able to update to modern Rubies, could we? First milestone was to have everything running on Ruby 2.3. What a victory that was, the impossible encoding issues were solved, not everything in a elegant way but still solved. and, what a relief. Now we were running on a Ruby version that is almost supported. Also the performance increase was unheard of. We continued the Ruby updates until Ruby 2.7, but that also a bit too late to actually be running on a supported Ruby version. Still the real beast to conquer was occasionally discussed but still avoided. Finally in 2023 there were time reserved and a "Rails upgrade" was scheduled to happen. How the heck do you upgrade a Rails version that is the age and size of ours, like how do you even start? Somehow we managed to pull it off. We sat down, closed the door and 3 months later we had upgraded from Rails 2.3 to Rails 7. The details and aftermath of this grind could be discussed at some other point.

https://pixabay.com/vectors/boat-travel-lake-pond-adventure-6517712/

So we are done, right? Well, not really. The main application that still powers most of our service got it's heart, brain and liver transplanted. But what about the other services created during the years of neglect and lost hope. All of a sudden they are more outdated than our original monolith. At this point I have to mention that not all of the services we have are a result of neglect and we are not going to just take everything we accumulated over 10 years and move back into the monolith. Until and beyond the day of the Rails upgrade was completed, dependencies were the concern of a few selected people that had the interest and knowledge. It's not like no updating happened during this 10 year history. But it was random and based on individuals and not all apps were as important to individuals as others. Still, even if it was fun at times I think no-one on the team would like to relive the journey of neglect again, so motivation to keep things in better shape was there, now it was time to put that motivation to work.

https://pixabay.com/vectors/family-childrens-day-6320382/

The idea of ownership started to grow and services were distributed among our 3 product teams sometimes based on logic and sometimes just a random pick. As mentioned some of the services created over the years were not created because of business domain but technical capabilities. For starters the responsibility was just to keep dependencies safe, so reacting on security issues in 3rd party dependencies reported by our security scanning tools. And being part of a bing corporation there were multiple tools to keep up with, and tools changing and so on. To help solve this problem of data spread over tons of different services we created a simple list that aggregates the information from different sources that was tool agnostic. Now every time a security bubble in our list turned red, we needed to act!

https://pixabay.com/vectors/eyes-surprise-wow-expression-open-312093/

The seed of ownership had been planted and some visibility added, now we can start expanding and growing it. But how to measure dependency freshness? "Our app is 2 major versions behind this dependency" does not sound that bad. Somehow someone was introduced to a concept called libyears, that measures the time between the version in use and latest released version. Makes talking about freshness on a totally different level. "Our app is 200 years old!" "All the apps we are responsible of are in average 100 years old". Sounds bad right? When we have something to measure lets make it visible and adjust the threshold, so Together we agreed on a baseline to aim at when it comes to dependencies. The goal and promise combined with visualizing the current state made a pretty powerful tool to motivate people like me that like keeping things tidy to be green instead of yellow. It's a nice feeling to be on the accepted level. Also showing all the teams libage in the same view can be considered a competition by some and therefore motivate on performing.

Dependencies are important to succeed, without Ruby and Rails I would probably not be here today But dependencies that bring value and make you fast can turn on you and tie you down and eventually slow you down Splitting into services should be done with caution and the right reasons. Lost hope is not a good enough reason. For us gamification and information radiators motivates teams to act