Patch Management
If there is one thing, a single thing, you can do to greatly improve your chances of fending off cyberattacks, it’s adopting a good digital hygiene strategy. In other words, adopting and implementing a comprehensive best practice methodology for updating all your information technology asset software and firmware running your hardware.
Updating your information technology assets is a function that is part of asset management and owned by your IT department.
Here, we will discuss the reality that updating your IT asset often takes second place to every other IT activities and by doing so, the company exposes itself to allowing attackers to take advantage of known vulnerabilities.
IT ecosystems are composed of many components; routers, switches, firewalls, desktop computers, servers, wireless access points, printers, wide range of software, operating systems and multiple network services, and many of these being private and internet facing.
Each and every one of these components require updating in order to fix bugs, add new features or address security vulnerabilities. This article will pay attention to the risks of updating these.
Out of the box vulnerabilities – Probably the best known vulnerability vector known to hackers are miss-configured IT assets. By this I mean IT administrator not taking proper steps to handle the security of the asset when it is first installed. Leaving the remote system administration IP and ports exposed to the user population or worst, to the internet, will allow hackers to attack this door and breach it. Isolating these remote admin service and whitelisting access to them is the first step to reduce risks. At the very least, you should have a vulnerability scan performed to identify the risks and help your IT staff address the potential issues.
The second best practice is to eliminate any “Default” configurations of IT assets. Information about default passwords, default administrator’s user name, default TCP ports, default profile names, and so on, information which will be released publicly and hackers use that information in attempts to infiltrate your assets. You would be amazed how many systems I accessed simply by knowing the manufacturer’s default configurations, as these are published the internet.
That is the reason why it is important to contain and control the remote access to the administrative services. I will admit that by experience and given only one choice, I would prefer to completely hide and isolate the administrative services access of an asset, rather than making it hard to access with a complex password. But doing both is the best practice.
Now, let’s discuss updating IT assets.
Asset inventory – The way to tackle updating IT assets is to first perform a detailed inventory. This task will allow you to sort the assets in order of priority and help you keep track of your progress. The inventory task should be handle by an IT staff, but identifying updating priorities should be performed by the a security professional.
I would recommend to the person who is handling the inventory schedule, to search the NIST.GOV website for inventory guidelines and sample forms. If you haven’t built an inventory list before, this is a good starting place for best practices and templates.
The inventory should cover the important details about every existing assets on your network, old and new. Installing a new system without first entering that asset’s details into the inventory list is a bad habbit and a risky one. If there are secret assets you don’t wan’t to put on a list, simply create a secondary list of secret assets, but keep those assets in inventory at all cost.
If your network is already fairly large and you are now just beginning to take inventory, there are tools that can help you scan your network, detect and list its assets. Some of my favourite tools are: Wireshark, NMap, InterMapper, PRTG. These are Active tools, meaning that they need visibility on your network and interact with it to discover network service nodes and make lists.
A more advanced tool, use to perform vulnerability assessments, is Nessus. It aggressively scans every IP and digs in each IP for details. Great tools to have if you are security professional. This tool is actively intrusive and should be used with caution.
Finally, Ricky Tan of Cyberspatial built and commercializing a passive scanning tool called Teleseer. This tool has the capability to build a layout of your network and all its assets simply by analyzing logs. Brilliant. This way, when network access is restricted, the tools doesn’t need to plug in it. It will just need logs, lots and lots of logs. Can’t wait to its release and see what it’s capable of.
Asset categories – Once you have created an IT asset inventory, you have to put each items in some sort of category of importance. Groups of criticality. This way, if you never have time to perform digital hygiene on all your assets, or have limited budgets to do this, you will at lease do it on your most important ones.
We can assume that you understand the difference between a critical asset and a non-critical one? An internet facing firewall is a critical asset. Carl’s bubble-jet printer in accounting is not.
The categorizing of assets can be done by your IT department, before the asset is deployed, and should be review by your company’s security governance committee.
If you aren’t familiar with GRC concepts (Governance, Risk and Compliance), I would recommend you take a master class. They aren’t difficult, don’t take a lot of your time, and will introduce a better way of organizing your thoughts about cybersecurity. Simplycyber offers a reasonably priced GRC training/certification program.
Update strategy – Here’s where things get a bit complicated. We agree that IT staff will perform the actual updating on assets. IT people would love to just go down the asset list and perform upgrades during office hours, but imagine the chaos it would create, specially with mission critical assets, like firewalls, routers or switches. IT will need assistance from the security governance group to identify maintenance windows to perform these updates.Story time: Many moons ago, I used to work at a company where Saturdays and Sundays were an open window for updates. There was nobody in the office and typically no one was working remotely over the weekend either. Update tasks were done fast and efficiently then regularly.
But years later, I worked in an industrial complex, with 7x24x365 operation and round the clock shifts. I will admit that updates were difficult to deploy. There was no communication channels between the IT staff and the company governance. The IT Director just left it to each staff to figure out how/when to perform maintenance. His instruction was to go and socialize the maintenance window to a dozen different directors/departments.I wish it would have been a bit better formalized. Updates was trying to herd cats.
Company governance need to pay more attention and pre-authorize maintenance windows for the IT staff, which will then simply announce upcoming work. The rest of the company will need to adapt their operations around the upgrade maintenance window. That’s an ideal strategy. But it will be up to you to figure our the strategy with an open discussion with the security governance group and all various department’s director. Just don’t leave the poor IT person in a lurch to figure this out by himself. IT staff already have their hands full researching, executing and in case of problems, reversing updates, within a limited time window. Help them avoid the company politics by doing that work ahead of time.
In a nutshell, your strategy is a general plan, should identify the WHAT and the WHY of your updating goals. Keep it simple, documented and add details to it later, and as needed.
Update Plan – A plan will cover the WHO and HOW of your updates. Once maintenance windows have been identified, the IT team will need to run thru the asset list and group them in priorities and build execution schedules.
I remember bringing up the subject of “frequency” of updates to a meeting involving many levels of governance. It was a disaster. I made my recommendations based on decades of experience, and no one payed attention to my plan. They were just too busy discussing the granular levels of self-satisfaction is we update less often, or not at all. Office politics will be your worst enemy, but if you can be a good diplomat, you can become an excellent security professional. It’s a job for “cool” demeanors. Hot heads shouldn’t become security professionals. In the example above, I leaned that blaming a governance group of not understanding my plan, was because I didn’t explained it properly. That’s when I started to adopt more visual aids. This approach helped a lot.
An good plan will identify the assets to be updated, who will perform the update task, the order it will be executed, the estimated time it will take to perform the update, the specific time an update is expected begin/end, fall back risks, contact information for each participants and idealy, as conference call number for participants to call into and coordinate the activity.
In case of reoccurring plans, pushing security patches to your desktops and laptops for example, the updates should be scheduled regularly, at the same period of the week and possibly with an announcement to the staff. This way, everyone knowing there is an ongoing update, and might avoid scheduling a very important presentation during that period of time. Inter-departmental collaboration and communication is important to maintain security. But rest assured, there will always be people criticizing any update plans you may present. To them, updates are disrupting and annoying. A security professional’s job, a CISO for example, is not to become popular, but to gracefully inform the people of those plans and putting emphasis on security. Nobody will complaint about having too much security!
In case of periodic updates, like a corporate firewall for example, an opportunistic maintenance windows can be identified and taken advantage of. A long weekend for example. Since these plans are more intrusive to the business operation, the security personnel (CISO) can help IT staff identify opportunities, but it still remains the IT department’s job to find a way to execute these updates.
What needs to be updated first?
Answer: The high risk assets identified on the inventory list.
How often do you update these high risk assets?
Answer: This isn’t a project, it’s a reoccurring operation to stay ahead of cyberattacks, vulnerabilities and all in all, anything bad that could bring the business operations to a halt. So, as often as possible is the best answer, but you need to take into account the business operation.
A security professional, or CISO, should be acquainted with all business functions in a company and the IT department’s capabilities. He’s the person who will listen thru the noise, will identify and then connect the proper dots to increase and maintain an acceptable level of security. A good CISO must have “thinking” and “resolving” skills to be efficient.
In case of security one alerts, when a really really bad vulnerability has been identified and a patch has been issued, scheduling an update might be a luxury. Often, a decision to enforce an interruption of service can happen depending of the risk assessment. In these case, an email underlining the event and importance of it can be send to the company staff maybe 15 minutes before the update. The CISO will need to be the judge and jury on these types of events and make decisions accordingly. Fortunately, these type of events are rare, once a year at most, but they need to be handled with care.
Cost – There is a cost to everything. The action of making IT staff work overtime, nights or weekends to apply updates and test, will have a cost to the company. The security specialist will need to evaluate the cost of performing the updates, versus, the potential cost of NOT doing the update. That’s a tricky equation, but if you need help in figuring out the cost of something, I refer you to the NIST.GOV website which holds a goldmine of excellent documentation.
Most cost will involve human resource time, and possibly material. Most vendors will offer software updates for free, especially security related ones. If you have a maintenance contract with any vendors, these update will certainly be included. If you have not kept up with maintaining your IT asset, it may increase the cost dramatically. The inventory list should include these factors to help you with cost assessment.
In event where critical security vulnerabilities have been identified and security patch availability have been announced, evaluating the cost is secondary to mitigating this vulnerability.
Example: The 2021 Log4J vulnerability affected many systems and the announcement of it should have motivated you to react immediately to mitigate it, without passing thru the usual approval channels. Let’s fix this and discuss later type of situation.
Designer Risk – It will be easier to implement and maintain security within your network if you create categories of assets, create networks for each categories and put relevant assets in each of hem. Then control how assets can communication to each others using firewall policies. This add multiple layers of security to your network and allows you to identify rogue players on your network using syslogs and monitoring.
I recommend you get familiar with the following government security documents: Zones management ITSG-22 (Or often referred to as Segmenting), and Risk management ITSG-33. These are definitely best security practices.
Once a proper network architecture has been taken care of, the update hygiene plans can be designed and implemented. Each assets will require an initial analysis to quality its importance, its security risk and the frequency of update it requires to keep up with vulnerabilities. Once you have a clear picture of assets, their risks and update requirement, you can build a plan and get IT to execute it.
Common Sense – In a high availability environment, the IT staff might only have 2 or 3 opportunities in a given year to perform some maintenance updates on critical equipment. The company mass storage array for example, or corporate VPN/Firewalls. But it has to be clear that zero day vulnerabilities need to be reasonnably addressed diligently and without delays. An emergency update schedule might have to be imposed on short notice. Make sure the company policies cover this kind of possible emergencies, to avoid confusion.
Update frequency – When being asked about how frequent should an asset be update with the latest security patch, my answer is always “It depends”. As a professional, I will maintain the bar as high as possible on assets I am responsible for. I also understand the business aspect enough not to become obsess with updating everything all the time, and causing daily interruptions of network services. You have to find the happy medium, a level of updates that will both satisfy the business operations, IT concerns, while maintain an acceptable level of risk.
Many large companies have prescribed policies to the effect of update frequencies. They did their due diligence in that regard and have documents which show exactly what needs to be done, how to do it and how often. Great! In this case, you simply need to follow the prescription.
In reality most companies aren’t large and lack the policies and documentation to cover cybersecurity. If they are lucky, these smaller companies will have sharp minded IT personnel who will take every available opportunities to apply updates. But that’s not the proper way to do this. C level executives needs to clearly understand and become owner of the risks associated with cybersecurity negligence, and the impact of that negligent should fall on the company governance, not IT staff.
It will be the CISO’s (Chief Information Security Officer) job to make all the executives aware of the risk factors. It will also be the CISO’s job to be on top of security exploitation news and trends, and finally it will be the CISO’s job to inform the IT department of all the incoming security threats.
I have seen too many times situations where C-Level executive think that ‘IT” is responsible for securing the company’s assets. That is a misconception. Cybersecurity can’t be imposed on a Database administrator or a System Admin or Telecomm staff. It’s the cybersecurity specialist’s job to operate the security radar, have insights about identifying and reporting risks to the upper echelons.
CISO – A CISO should exist to bridge the gap and build communication channels between technically agile people (IT/OT), and the executives running the business aspect (C-Level). A CISO’s job is to interrupt people’s daily lives, identify their bad technology hygiene habits, and to build a clear blueprint of how to address risks.
Perfect security does not exist and will never be achieved. A CISO will facilitate targeting the highest levels of security to which the company agrees with. In some cases, the company might decide not to invest in cybersecurity and spend the money elsewhere. That strategy is a business decision. IT staff who aren’t getting budget to operate a proper cybersecurity department, or are understaffed, shouldn’t feel bad if they get attacked and breached. It’s the decision maker’s fault.
Updates Management – Most modern information technology software and hardware have ways to getting updates automatically. If they don’t, they offer centralized management tools that will help with this task.
Updating a large IT ecosystem composed of thousands of laptops and desktops isn’t done manually. In the case of Microsoft Windows, the Active Directory policies (GPO) will control the flow and frequency of updates. The addition of a WSUS (Windows Update Server) to the network will help with pre-downloading various types of updates, sort updates by importance (Security, Features…etc), place all laptops and desktops into various categories (HR, Accounting, Finance, Engineering, Sales..etc), and then control the update release to various categories of assets.
User desktops/laptops should be update weekly because they expose themselves to the internet daily and browser features are constantly being exploited for vulnerabilities. Servers should get updated no less than 4 times per year. More often if they are exposed to the internet. Network assets should follow the same frequency of updates as server, but should be updated more often if exposed directly to the internet (Firewalls for example).
With the ever increasing numbers of vulnerabilities being found and the numbers of attacks following, the frequency of updating can only accelerate. This is due to the increasing complexity of systems and software. Being on the verge of a world war also demands increased vigilance, increased updating frequency and hardening of assets.
Systems VS Subscriptions – You shouldn’t confuse updating an operating system with a updating a subscription service. A subscription services contains signatures used by a specialized software to perform a security task. An anti-virus software for example will need to update it’s library on daily/hourly base and retrieve information about the most recent vulnerabilities it scans/protects for. A firewall might have an IDS (Intrusion Detection) or IPS (Intrusion Protection) service and need to be updated frequently, every 4 to 12 hours for example. Your update strategy should account for these services and they should be identified in the inventory list under the “service” category.
Updating the system’s software (firmware) involves replacing the whole operating system of that asset. This can have major impact if the update is release and hasn’t been tested thoroughly. And advise I can give is to wait a few weeks before deploying major OS updates, and wait until rest of the community comments it. Then, you can deploy updates on secondary systems, test systems, or smaller groups. Then decide on deploying to larger groups if no adverse effects have been reported.
Conclusion – In the past, I have rarely used a written strategy and plan to perform updates around the network environment I managed, and that’s a bad bad thing. I got away with it because I quickly memorized every assets and can quickly make decision about update requirements. I often was a one-man-band in the early days, and documenting something I already knew was a waste of my time. But the downfall is that if I get hit by a buss, there is no legacy documentation, and Junior IT staff have no more access to my knowledge base.
Today, you need written policies, strategy and plans. Don’t be afraid of doing them, they don’t have to be 200 page documents. A one or two page document describing the who, what,when,why of applying updates to your IT assets might just be good enough to start with. But I would recommend investing more time and energy on a good inventory list of all your assets. That will become an essential tool not only to IT, but for the whole business.
As a side dish to the inventory list, I would recommend you create and maintain a comprehensive network layout/diagram of how your assets are distributed on your network. My skills allowed me to memorize complex networks and be able to quickly draw any portions of it as required. But when you need to explain to decision makers the reasons to invest in cybersecurity, you’d better have a nice and detailed diagram to show. A wise man I worked for once told me “Don’t tell me about it, just show me”.
The last few years showed urgent needs for better security. Attacks are getting more sophisticated, solutions to protect against them are getting less and less efficient. This is due to the increasing complexity of software and the ever increasing use of encryption reducing the possibility to detect signatures during transport.
But definitely, the technology industry is making gargantuan efforts to keep up with the security playing fields and their efforts are only accessible thru updates they make available. If you don’t update accordingly, you will fall behind and increase your chance of becoming a breach statistic.
Luck will favour the prepared.
This was my view.
Cheers & Beers.
