“Software is now nationally critical infrastructure for most countries. For banking, energy, insurance, pharmaceuticals – even gaming, which is a very sophisticated industry these days – software security is going to be a top three priority over the next decade. And regulators are now paying attention because after the Colonial Pipeline incident they realised that it’s not just a ‘techie’ issue when your oil supply stops.”
CloudBees CISO Prakash Sethuraman, a former global head of cloud security at HSBC who was appointed in June 2021, is building up a fairly impassioned head of steam, sitting with The Stack in London. Colonial Pipeline’s breach started with a compromised password on a VPN account – an issue of credentials management and poor security hygiene rather that something broken in a software build – but it's a moot point; Sethuraman is right: regulators have sat up and started paying attention to not just cybersecurity broadly, but software security.
US regulators are working on plans [pdf] to mandate a detailed Software Bill of Materials (SBOM) for example so that they can quickly understand the building blocks and dependencies of software running critical computer systems. The scramble to understand dependencies after the Log4J incident – when a vulnerability in a widely used tool in the ubiquitous Java ecosystem exposed swathes of widely used enterprise software applications to easy exploitation – only focussed minds more. The White House's recent Software Security Summit meanwhile gave futher momentum to a range of software security efforts involving the public and private sectors.
See: Global CISOs, White House, agree 10-point OSS security plan
“This is an inflection point in the industry,” claims Sethuraman.
“Software, how you build software, how you automate the building of software, how do you make sure that that automation doesn't introduce risks in the process, and it continues to be secure; that's going to be a key topic for not just CISOs but enterprises more broadly over the next five years,” says the CloudBees CISO.
“Because most organisations like banks, pharmaceutical companies, ecommerce companies, they write about 25% to 30% -- if you're really generous -- of the code that their applications use; most of the applications are composed of third-party products, open source libraries, etc. and it remains true that most people use this stuff without understanding the complex chain of dependencies."
"The guy who changed one npm library and caused chaos* proves that point" he adds "and in a complex ecosystem you have to automate to be able to deal with that complexity.”
"Now the conversation has moved to how do you secure applications from the ground-up, because that's where the problem is now... The supply chain is not [just] an attack surface problem; the supply chain is a 'how do you build the damn thing in the first place?' problem. You don't wait for the CISO to come along and say 'that thing that you designed and built won't work, because you will get attacked every day'," he emphasises.
Join your peers following The Stack on LinkedIn
CloudBees, where Sethuraman combines a CISO role with ownership of the company’s compliance product, was founded by former JBoss CTO Sacha Labourey in 2010 as a provider of enterprise support for open source Jenkins; a server-based system for building, testing, and deploying software. It has since evolved considerably.
It now offers a suite of tools designed to help enterprises connect, automate and orchestrate tools across development, operations and shared service teams to optimise software delivery. The idea is to have all of its software delivery automation tools available as a single unified product under a “software delivery management” rubrik: from continuous integration (CI) to continuous development (CD) via application release orchestration (ARO) -- a set of overused acronyms that we'll come back to and contextualise further below.
(As Labourey earlier put it to The Stack in an earlier conversation: “As organisations go faster and start pushing changes every day, every hour, you need a ‘brain’ of software delivery, that’s able to capture all of the signals that are taking place, in your Jira, in your Git and your Jenkins, and in production; that's CloudBees.”)
Hit me with your jargon stick
The concept of CI/CD remains an opaque one for many not close to the developer worksurface and even there, sometimes, runs into puzzlement. So what the hell are CI/CD and ARO and why are they becoming increasingly important for software security as regulators start sniffing about enterprises look to reinforce the security of their applications?
CI, briefly, means regularly adding verified changes to the codebase of an app and more frequently integrating them with that app, without hiccups. CD means automatically and continuously checking that the code as been approved for release, so that new features are production-ready.
As Sethuraman has written: "The fear of deploying a breaking change into production causes many growing companies to create unnecessary processes, strict approval gates and even deployment committees or boards in an effort to prevent negative impact to customers. In the enterprise, deployments tend to be more complex and they must often adhere to rigid compliance, governance and regulatory audit needs. Building a CI/CD strategy that connects engineering teams to the stakeholders requesting and approving their changes improves trust and increases the velocity at which new and exciting features can make it out to customers..."
CloudBees CISO on lessons from HSBC
Writing secure code and building secure applications is a challenge exacerbated by both scale (at bare minimum owing to lots of pods of developers writing code for lots of applications) and an environment in which many large enterprises are on the much-discussed journey from monolithic, on-premises architectures to cloud and microservices-based systems. Sethuraman has seen the challenges that have emerged around this up close, not least as Chief Architect (Digital Technologies) then Global Head of Cloud Security at HSBC.
Pressed for lessons from that period of his career he’s circumspect – HSBC is one of CloudBees biggest customers after all – praising the bank’s Group CIO and leadership for working hard to introduce a more Agile approach as it continues to go through a “not trivial transformation”, before sharing some considered views.
“For an organisation with the level of resources of an HSBC, money is not the issue” he notes.
“The change you're contemplating and the complexity of the change you're contemplating is the issue… Even with resources and with backing from the business [digital transformation] is a challenge and getting it done on the ground is a completely different kettle of fish [to envisioning it]."
"Human beings are not very good at dealing with complexity" he adds: "Trying to get people to align and make a decision takes time [but] meantime-to-decision is a fundamental part of agility… cloud comes with a lot of choice and with choice and flexibility, comes complexity of decision making [that is uniquely challenging] when you have a large number of people that you need to take on a journey with you.”
Pivoting to a container-centric approach at scale is a particular cultural challenge, he tells The Stack: “Most large organisations have gotten used to dealing with applications that number in the low-to-medium thousands, to 10,000, to maybe 15,000 at really diverse organisations. Human beings are used to dealing with support tooling to deal with applications that are often fairly static for a long period of time.
"Now we are in the world of cloud, where things spin up, spin down, so they've got fluidity – the word 'ephemeral' is kind of overused – but things change all the time. Suddenly, you've gone from 200 virtual or physical servers that might be running an application, to 20,000 containers running that same application. For people to wrap their heads around this conceptually, visually, and to be able to just contemplate that scale is difficult: all that is a long way of saying it [building and running applications] has got a lot more complicated!”
https://www.youtube.com/watch?v=CUHrjT_gtgY
CloudBees' customer base is primarily bluechips tackling problems at scale. Over the past few years, many have been under pressure from upstart startups and other disrupters to innovate faster, but Sethuraman is sympathetic to what this looks like at scale, noting: "A lot of articles are written about 'how do you build this stuff?' but building the stuff is only 1/10 of the problem. You've got to keep it running, perhaps for five years, or 10 years.
"Not enough attention is given to what will you do when things don't work; when there's a production incident, how are you going to address all this complexity and figure out what went wrong?
"What a lot of people forget when they talk about CD and then they talk about deployment in general [is that] for anybody who's been there, trying to deploy software at scale in a large organisation, your question is never 'how are you going to deploy this?' Your first question is 'how are you going to back it out?' Because stuff happens, things don't work, things always don't go to plan. You cannot turn up at 3am and try to invent a backup plan -- that doesn't work. So part of release orchestration is to know what the previous state was, what the new state is, and therefore what steps do I need to take to back things up.
"We now are talking about progressive delivery, which basically says when you write software, make sure it's in your production environment" he says, hastening to add that "just because a binary is in your production environment doesn't mean users have to be exposed to it, I can selectively choose to expose something to you and the same binary behaves differently for someone else. That is 'feature flagging', as a capability that allows me even in production to control my risk radius... You don't want something not working for 10 million people; rather test it on 100 people and then scale up." (Feature flags are switches that let you change application behavior in runtime without redeploying; yes, CloudBees offers an enterprise system to do this. By enabling or disabling a block of code, they minimise the risk of introducing a new feature into a build.)
Prakash Sethuraman: You need to assert compliance in real-time
Stepping back from this, the roots of insecure software and architectures run deep, he suggests: “In my first job, I was writing application code on an ICL VME; I was also the sysadmin for a mainframe. So short of laying the cables – it was a factory, they wouldn’t let me – I was doing pretty much everything. That's where the industry started.
"Then it went through this phase of specialisation to the extent that most of the application engineers today don't necessarily understand how the hardware works, how networks work, how data is transmitted. That discontinuity between individual specialisations is what the bad guys look for; because the left hand and the right hand not talking to each other, that lack of a holistic view of security, introduces gaps they can exploit.”
As a result, sophisticated automation that can identify those gaps -- particularly in application builds -- is critical he says, suggesting that key CISO priorities should be to "get your organisation to start thinking of security as an ecosystem. So it's ecosystem security, versus point-in-time, point-in-place kind of security".
"The software supply chain, and the security of your software supply chain has to be top on your priority list [and] process validation and certification of processes vis-a-vis security and compliance is not adequate anymore; you have to be able to assert compliance in real time. These are all part of how you think of your secure ecosystem.
"The second thing to really think about, is in a world where software determines your ability to compete, introducing friction to your developers through your tools or processes is not OK, so always put the importance of minimising friction on top of your agenda," he concludes: "The federation of security expertise is not easy -- so you have to consciously think about that and broaden the base that worries about security within your organisation."
*Ed: Sethuraman is referring to the 2016 deletion of an overlooked 11-line npm package that when unpublished broke JavaScript-based applications around the world, detailed by Quartz here.
