Everyone needs effective patch management. This critical but tedious process secures organizations of all sizes by eliminating vulnerabilities and delivering product upgrades.<\/p>\n\n\n\n
Patching requires urgency. Attackers begin to reverse engineer patches immediately to exploit unpatched systems, even as organizations can leave known vulnerabilities unpatched for years. A Ponemon Institute survey of breach victims found that 60% of attacks<\/a> exploited known vulnerabilities that had patches that hadn’t been applied yet.<\/p>\n\n\n\n Patching remains a problem because the details of execution can overwhelm an organization. Best practices for patching apply to organizations of all sizes and can improve time-to-patch, but the implementation will vary enormously depending on resources.<\/p>\n\n\n\n Best practices for patch management include:<\/p>\n\n\n\n This article will explore each practice in detail to explain each role, minimum expectations, and provide tips to help make a smooth process. For a quick review of the topic, see: What is Patch Management?<\/a><\/p>\n\n\n\n To know what to patch, an organization first needs to know what it has. This is more formally called Asset Discovery, which the NIST Cybersecurity Framework, CIS Critical Security Controls, and other frameworks use as the foundation on which to build a strong cybersecurity program.<\/p>\n\n\n\n Use automated tools.<\/strong> A clipboard and paper or excel inventory only works for the very smallest organizations. Even modest-sized organizations adopt automated tools to scour the network<\/a> for every device, endpoint, server, operating system (OS), and application. Yet beware, some automated software may only track hardware, so a patch management team needs to understand tool limitations.<\/p>\n\n\n\n Track critical assets. <\/strong>If the failure of the asset can disrupt the business or cause costly data breaches, track it. Most organizations track all endpoint terminals (PCs, laptops, tablets, mobile phones, etc.), servers, cloud resources (containers, repositories, and other in-house managed infrastructure), critical software (accounting software, HR software, etc.), and network equipment used regularly.<\/p>\n\n\n\n Track all assets<\/strong>, not just computers. Larger organizations understand the risk of ignoring the internet of things<\/a> (security cameras, Wi-Fi enabled TVs, etc.), operational technology (wi-fi controlled pumps, temperature sensors, etc.), medical technology<\/a> (especially wireless devices), point-of-sale (POS) systems<\/a>, and rarely used assets.<\/p>\n\n\n\n Track app assets. <\/strong>Asset tracking also applies to applications developed in-house. While this will probably be managed by a separate DevSecOps<\/a> team, the chief security officer should still receive reports regarding the software bill of materials (SBOM)<\/a> and the status (version, last update, etc.) of application components.<\/p>\n\n\n\n Anticipate exceptions<\/strong> that may require special controls or planning. For example, in a hospital, the patient record database cannot go down and the Windows 95 machine controlling the MRI machine cannot be upgraded. These will need to fall outside of the standard processes for special handling or extra security controls.<\/p>\n\n\n\n Also, users need the ability to delay patches temporarily. No IT team wants to explain to the VP of Sales why mandatory update restarts disrupted a very important presentation. However, IT teams also need the ability to force restarts for employees that continuously push off updates.<\/p>\n\n\n\n Effective asset management tools<\/strong> can often identify the current versions of the software or firmware. This will make it easier to track assets and determine if patches or more current versions may be available.<\/p>\n\n\n\n Standardized assets<\/strong> can reduce the burden on the IT team to track assets and their current status. Many organizations replace PCs in larger batches or limit software installations to reduce the number of hardware and software configurations to track and maintain. Organizations centralizing control or executing a merger will often inventory software categories (accounting, HR, etc.) and switch all users onto a single software package to help simplify updates (among other benefits).<\/p>\n\n\n\n Place assets in context<\/strong> using various network maps and diagrams to identify network segments, networking equipment, firewalls, and security controls that protect each asset. This context will play a role later in determining the overall risk of unapplied patches.<\/p>\n\n\n\n Quarantine unpatched unauthorized and BYOD devices<\/strong> that fall outside of the organization\u2019s patch management control. Network access control (NAC) solutions<\/a> can detect and quarantine devices that lack security patches and updates to protect the network.<\/p>\n\n\n\n Also see:<\/strong> Top IT Asset Management (ITAM) Tools for Security<\/a><\/p>\n\n\n\n Risk management requires an organization to understand the value of an asset (device, database, application, etc.) and the value of data contained in, secured by, or flowing through that asset. Risk should also reflect the damage to the organization if the asset fails or becomes compromised.<\/p>\n\n\n\n Start with the most important. <\/strong>At the very least, risk management profiles should be in place for the most valuable and operationally critical infrastructure most likely to be targeted or affected by an attack. The number of assets that can be tracked may depend upon the resources of the organization and its ability to use the information effectively.<\/p>\n\n\n\n Characterize the likelihood of an attack<\/strong> based upon the security and operational controls defending the asset. Ratings can be categorical such as High\/Medium\/Low or rated 1\/2\/3\/4 and these ratings should be used to prioritize the importance of patching the asset. For example, a data storage server rated medium value and low risk of attack will be of lower priority than the customer application server rated of high value and high risk receives its patches.<\/p>\n\n\n\n Use <\/strong>risk management software<\/strong><\/a> to manage and maintain current risk profiles of tracked assets. The better software will enable an enterprise to characterize the risk in currency (US dollars, Euros, etc.) to help justify security spending to protect the asset. Ideally, the risk management, asset management, patch management and other related software can interconnect to share information and enable real-time analysis, automated prioritization, and alerts.<\/p>\n\n\n\n Group assets<\/strong> by risk class to help manage large numbers of assets, enable operational efficiency, and effective prioritization. A categorical asset class can speed up the assignment of a risk to an asset. However, this technique is best for large numbers of non-critical assets (workstations, switches, etc.). Key assets should be individually assessed.<\/p>\n\n\n\n Segregate patch-exempt assets<\/strong>. Not all assets can be patched. While upgrades may be preferred, sometimes it isn\u2019t possible so the devices need to be segregated. For example, hardcoded credentials<\/a> in IoT devices (especially medical technology) may require those devices to be placed on a separate network segment with a whitelist device list or similar security controls that protect against access from unauthorized access.<\/p>\n\n\n\n Harden security by default.<\/strong> Many organizations allow vulnerabilities and open ports to remain on assets perceived to be buried within the network behind firewalls and other controls. However, zero trust<\/a> philosophies require organizations to assume compromised networks and the frequent effectiveness of ransomware attacks<\/a> indicates this assumption will be accurate more often than an organization might like. <\/p>\n\n\n\n Disabling unneeded protocols and unnecessary ports to harden assets will reduce a potential attack surface. Hardened infrastructure also lowers the overall risk of the organization – and reduces the urgency for some patches.<\/p>\n\n\n\n Context matters. <\/strong>Risk scores should incorporate the results of current vulnerability scans, possible public access (internet connections, physically accessible in a public place, etc.), and known vulnerabilities left unpatched.<\/p>\n\n\n\n Upgrade and retire legacy technology.<\/strong> Some technologies, especially in hospitals and for industrial applications, enjoy a lifespan that outlives the operating system required to manage the technology. MedTech and OT is notorious for requiring the maintenance of Windows 95, Windows 7, and other obsolete operating systems that can no longer be patched.<\/p>\n\n\n\n The cost to protect these systems continually increases both in hard dollars as well as the labor cost of the IT security team to implement controls. Organizations need to accurately assess these costs and invest in replacing obsolete technology.<\/p>\n\n\n\n Keep in mind that some upgrades may be possible that accomplish the same goals. For example, an obsolete Windows 95 computer can often be replaced with a fully updated Windows 11 machine that can host a secured virtual environment with a Windows 95 emulator.<\/p>\n\n\n\n Blocking potential attack paths for assets that can’t be patched or haven’t yet been patched is another defensive tactic, a process known as virtual patching<\/a>.<\/p>\n\n\n\n Account for legal risk<\/strong>. When calculating the risk of an asset consider the legal liability for the organization if an adversary gains control. What would the legal liability cost the organization if the asset was unpatched and a judge, jury, or regulator found the organization to be negligent?<\/p>\n\n\n\n See the <\/strong>Best Patch Management Software & Tools<\/strong><\/a><\/p>\n\n\n\n Many IT professionals seem allergic to documentation. However, documentation helps to set expectations, track status, and communicate clearly within the patch management team and to other stakeholders.<\/p>\n\n\n\n Adopt a basic <\/strong>patch management policy<\/strong><\/a> to establish baseline requirements for what to patch, patching process<\/a> details, how to manage exceptions, and how to manage disruptions. A few weeks of negotiations can prevent more permanent grudges between departments.<\/p>\n\n\n\n Schedule patching and downtime<\/strong> to minimize impact and disruptions to operations. Determine the approval process, notices (internal, customer, etc.), approvals, and other documentation required when assets must be disabled or restarted to apply patches.<\/p>\n\n\n\n Generate reports<\/strong> to track what assets were upgraded\/patched, exceptions, and any issues encountered. Good documentation makes the next patching cycle more efficient and helps prevent confusion during IT team transitions or incident response.<\/p>\n\n\n\n Assign ownership<\/strong> of patch management to an executive or manager and make it a key performance indicator. The process can be delegated or even outsourced, but ensure the designated executive or manager remains familiar with the results and the process.<\/p>\n\n\n\n Patch management team definition and coordination<\/strong> is required at the enterprise level due to extra complexity. Patching may be performed by multiple patch management teams, cover diverse IT environments, and be forced to meet multiple compliance standards. Better patch management tools can help coordinate activities, classify assets, embed workflows, and enable comprehensive reporting.<\/p>\n\n\n\n Change management<\/strong> policies and reports will be needed to establish who notification requirements, how specific changes should be documented, what reports should be generated afterwards. Change management reports help to verify approved changes to critical assets as opposed to unauthorized and potentially malicious changes.<\/p>\n\n\n\n Be detailed.<\/strong> Specify the members of the patch management team and their responsibilities clearly. Determine required notification processes in detail. If different processes or teams will be required for specific technologies (servers, cloud platforms, etc.) create exhibits with specific teams and requirements. However, be careful not to be so detailed that the IT team cannot flexibly make adjustments according to available resources and circumstances.<\/p>\n\n\n\n Publicize patching processes<\/strong> to make regular patching processes known and predictable for the organization. Users get used to the regular process and understand that normal patching isn\u2019t typically disruptive or burdensome. This also adds credibility to urgent patches by making them seem even more important.<\/p>\n\n\n\n Take patching seriously<\/strong> and require the reporting of key metrics at the executive level such as the percent of devices patched, the time to patch for critical and non-critical patches, exception reports, etc.<\/p>\n\n\n\n The key goals for patch management often conflict because quick application of patches to block security vulnerabilities can introduce disruption to operations and require already-committed IT resources. An effective patching process manages these conflicts to obtain the correct patches, ensure their effectiveness, and deploy them with minimal operational impact.<\/p>\n\n\n\n Monitor for updates<\/strong>, especially the most critical assets. Many vendors offer email or similar mechanisms to notify teams of updates; for example, almost all organizations should directly or indirectly subscribe to Microsoft Technical Security Notifications<\/a>.<\/p>\n\n\n\n Only use verified sources<\/strong> to download patches and updates. Some malicious hacking groups create websites and phishing attacks offering security updates embedded with malware.<\/p>\n\n\n\n Automate where possible.<\/strong> Small IT teams can be overwhelmed with the volume of updates on top of their day-to-day duties. Various vendors provide patch management tools that automate patching for specific assets (Windows machines, macOS, third-party software, networking equipment, etc.). Some vendors also offer free tiers for smaller companies. Before adopting a specific tool, verify what assets will be covered and test the tool to verify it will relieve and not burden the patch management team.<\/p>\n\n\n\n Some small businesses rely upon Microsoft\/Apple automated patch management built into the operating system, but they need to recognize two key weaknesses. First, the organization will not have visibility into the status of the endpoint or when patches may fail or be delayed by users. Second, the OS may be a critical asset, but it is hardly the only asset that needs to be updated on a regular basis. An OS-centric update will neglect vital updates for Adobe, Google, Mozilla, Oracle, and many other vendors and applications. A more formal and centralized patch management tool or service is a superior option.<\/p>\n\n\n\n Prioritize patches<\/strong> based upon the risk of the asset and the criticality of the vulnerability being patched. Non-critical patches should be updated on a scheduled basis to minimize disruption.<\/p>\n\n\n\n Coordinate with operations<\/strong> or other stakeholders and follow careful communication and pre-approved procedures for patches that will cause operational disruptions.<\/p>\n\n\n\n Audit assets to locate failed patches<\/strong> or to recover devices or assets disrupted by an applied patch.<\/p>\n\n\n\n Report results<\/strong> as required by the patch management policy (see above) or basic KPIs. Update the asset list with the latest software\/firmware versions to make future patching processes more efficient.<\/p>\n\n\n\n Threat intelligence<\/strong> and government resources such as the NIST National Vulnerability Database<\/a> should enhance vendor monitoring for vulnerability and patch announcements. These feeds provide additional context regarding actively exploited vulnerabilities and helps to prioritize patches that may be actively exploited or especially critical.<\/p>\n\n\n\n Leverage vendor relationships<\/strong>, register devices, and keep in contact with sales reps and suppliers to receive patch notices as quickly as possible. Some vendors have been known to provide direct notice to customers in advance of public vulnerability and patching announcements.<\/p>\n\n\n\n Centralize patch management<\/strong> to consolidate expertise, reporting, and eliminate redundancies (repetitive testing, waiting for redundant rollout, etc.). Centralized patch management can also create local network distribution points to reduce network and internet traffic from thousands of machines reaching out to a vendor\u2019s site to download patches.<\/p>\n\n\n\n Patch priority<\/strong>, relevance and urgency can be more difficult to determine in enterprise environments. Diverse technology, version inconsistency between offices, and variations in the tech stack complicate the understanding of risk and how vulnerabilities might apply.<\/p>\n\n\n\n \u201cIt is hard for many enterprises to accept the fact they cannot patch everything,\u201d explains Bob Kelly, director of product management at Flexera. \u201cAs little as one in 10 patches actually get deployed due to challenges in identifying, testing, and rolling out updates to the constant backlog of security updates coming on a daily basis.”<\/p>\n\n\n\n A patch management solution that integrates with risk values, threat intelligence, and CVSS 3.1<\/a> can help to automate or at least rapidly understand the risk profile of the vulnerability. Also, keep in mind that unapplied patches may be replaced by subsequent patches. An organization will need to make sure that obsolete patches are dropped from the queue.<\/p>\n\n\n\n Verify security control assumptions.<\/strong> Controls can mitigate vulnerabilities and decrease urgency, but those controls must be verified. For example, verify there is no overlooked access to a critical vulnerability on a database server that is supposed to be only accessible on an internal network. Bad assumptions can undermine the understanding of risk and leave critical assets more exposed than expected.<\/p>\n\n\n\n Test patches<\/strong> in a test environment to check for conflicts with other software, hardware, and IT architecture. If resources are constrained, focus on critical patches and critical systems. Testing is typically out of reach for smaller organizations, and can cause critical delays for high-risk patches.<\/p>\n\n\n\n Fortunately, many patch management tools and service providers<\/a> include patch testing as part of their package – with some testing patches within hours of their release. While pre-tested patches may eliminate the risk of broad disruption, keep in mind that each organization’s architecture and assets will be unique and may experience unusual circumstances.<\/p>\n\n\n\n Limited patch rollout<\/strong> to specific devices. Staggered deployment allows the patch management team to observe results and lower the network bandwidth requirements of large scale patch deployments. Limited deployment can also limit the damage of unexpected conflicts from a patched vulnerability.<\/p>\n\n\n\n \u201cApplications and computer systems are extremely complex, and there\u2019s always a chance that a new update may create unintended problems,\u201d cautions Lou Fiorello, vice president and general manager of security products at ServiceNow. \u201cRoll out a new patch in a controlled environment before trusting it with an entire network.\u201d<\/p>\n\n\n\n Next level-automated tools<\/strong> expand patch management to networking equipment, servers, IoT devices, and a wide variety of third-party software. However, different tools will have different management and integration capabilities so patch management teams may need multiple tools and still have some assets that require manual patching because they are not covered or because they require manual processes.<\/p>\n\n\n\n Redundant and failover systems<\/strong> provide a built-in test environment for patch deployment and for less disruptive patch roll-out. A web server\u2019s failover clone can be patched and checked for potential conflicts. Once verified as successfully updated, the failover server can be deployed as the primary server and the former primary server can be updated and put into service as the failover system. Cloud, containers, and SaaS technologies often use this technique to maximize uptime.<\/p>\n\n\n\n Extend patching to AppBoM.<\/strong> Larger organizations with application development need to monitor open source code and libraries in their application bill-of-materials. The DevOps or DevSecOps teams may have an independent patch management process, but their own patch management reports should be reported to executives either separately or in a consolidated report.<\/p>\n\n\n\n Patch with urgency.<\/strong> Mandiant estimates that more than 27% of disclosed vulnerabilities are exploited less than a month<\/a> after the release of a patch. Lower risk systems can be patched with less urgency, but keep in mind that zero-day attacks<\/a> or other undetected vulnerabilities can allow attacks on those lower risk systems to jeopardize systems of much higher value.<\/p>\n\n\n\n Patching priority does not equate to risk value. <\/strong>Some organizations exclusively patch Common Vulnerability Scoring System (CVSS) vulnerabilities with a score of 7 or higher, but attackers take advantage of this trend. \u201cWe are seeing an uptick in exploitation of vulnerability in the 5\u20137 range because hackers know that those scored 7 and higher are more likely to be mitigated faster,\u201d warns Bob Kelly, director of product management at Flexera.<\/p>\n\n\n\n Organizations need to combine CVSS scores with risk ratings and known exploits. The U.S. Cybersecurity and Infrastructure Security Agency (CISA) maintains a list of known exploited vulnerabilities<\/a>. The list currently only shows about 900 vulnerabilities \u2013 a seemingly small number when more than 20,000 new vulnerabilities are identified each year \u2013 but it still requires good asset discovery and prioritization to target just that relatively small list.<\/p>\n\n\n\n Combining risk management, CVSS scores, and active exploitation will create a patching priority such as:<\/p>\n\n\n\n Hire expertise or outsource<\/strong> if patch management goals cannot be met using in-house resources. Patch management is one of the most widely outsourced functions for managed IT service providers<\/a> (MSPs) who can patch a wide range of systems and software for competitive prices. Before signing a service agreement, companies should verify that the MSP will meet or exceed the patch management goals for patch deployment speed, minimized operational disruption, and reporting.<\/p>\n\n\n\n Also read:<\/p>\n\n\n\n Even the best plans, processes, and technology can fail. Organizations need to maintain disaster recovery plans and processes to recover systems in a broad range of scenarios. The elements of disaster recovery for patch management revolve around restoring systems and data.<\/p>\n\n\n\n Backups<\/strong><\/a> <\/strong>should always be executed prior to patching to enable effective roll back or back-out of updates. Local system backups often will work, but in the event of a catastrophic patch failure, independent backups on different assets will be needed.<\/p>\n\n\n\n Vulnerability scans<\/strong><\/a> should be conducted after patching to verify proper installation and to check for potentially undisclosed issues. Some patches have been known to cause conflicts with security tools or introduce new vulnerabilities.<\/p>\n\n\n\n Periodic <\/strong>penetration tests<\/strong><\/a> should be conducted to verify that the controls protecting unpatched systems remain effective. Risk assessments assume controls continue to work; however, these assumptions should be tested.<\/p>\n\n\n\n Also read:<\/strong> 8 Best Disaster Recovery Solutions<\/a><\/p>\n\n\n\n Organizations that execute patch management effectively will enjoy many benefits such as improved security, improved features, and avoiding costly incident responses. Effective patching policies prevent far more costly emergency IT responses that force overtime and additional costs for asset recovery.<\/p>\n\n\n\n More importantly, breaches can also trigger compliance violations that lead to lawsuits and regulatory fines. Many governments such as Australia<\/a> and the United States<\/a> list vulnerability patching as fundamental IT security practices so failure to patch will likely be viewed as negligence and punishable by stiff penalties.<\/p>\n\n\n\n Patch management can be boring because it is so fundamental. However, execution can be complex and many breaches reveal that many struggle to manage their processes well. Best practices can enable effective implementation of the patching process and should be adopted by organizations of all sizes.<\/p>\n\n\n\n Read next:<\/strong> Top Vulnerability Management Tools<\/a><\/p>\n\n\n\n This article was originally written by <\/em>Drew Robb<\/em><\/a> on October 21, 2022. It was updated by Chad Kime on February 24, 2023.<\/em><\/p>\n\n\n\n
1. Asset Management<\/strong><\/h2>\n\n\n\n
Minimum Expectations<\/h3>\n\n\n\n
Next Level Implementation<\/h3>\n\n\n\n
Pro Tips<\/h3>\n\n\n\n
2. Risk Management<\/strong><\/h2>\n\n\n\n
Minimum Expectations<\/h3>\n\n\n\n
Next Level Implementation<\/h3>\n\n\n\n
Pro Tips<\/h3>\n\n\n\n
3. Documentation<\/strong><\/h2>\n\n\n\n
Minimum Expectations<\/h3>\n\n\n\n
Next Level Implementation<\/h3>\n\n\n\n
Pro Tips<\/h3>\n\n\n\n
4. Effective Patching Process<\/strong><\/h2>\n\n\n\n
Minimum Expectations<\/h3>\n\n\n\n
Next Level Implementation<\/h3>\n\n\n\n
Pro Tips<\/h3>\n\n\n\n
\n
\n
5. Mistake Management<\/strong><\/h2>\n\n\n\n
Minimum Expectations<\/h3>\n\n\n\n
Next Level Implementation<\/h3>\n\n\n\n
Pro Tips<\/h3>\n\n\n\n
Bottom Line: Patch or Pay<\/h2>\n\n\n\n