To Self-Assess or Not to Self-Assess: The Security Dilemma

To maintain a well-protected security program, best practice stipulates assessments of security capabilities should be carried out at least annually, emphasizing continuous improvement. But as security professionals, we know firsthand that not all budgets are created equal. So, when it comes time to crunch the numbers, suggestions like foregoing a $60,000 third-party assessment for a self-assessment seem logical, especially with the availability of generative artificial intelligence (GenAI), right? Well, not exactly – let’s discuss.

Accountability

Despite the likes of documented roadmaps, Gantt charts, and other project management-related tools, the day-to-day of a security professional is unpredictable. Between alert triaging, patch implementation, policy development, and training activities, kicking the can down the road helps to declutter minds and desks.

But as the road stretches further into the distance, activities like self-assessments fall by the wayside. Third party assessors assist with accountability – and apparently, alliteration. Statements of work (SOWs) bind organizations to the performance of an assessment, ensuring it will be completed within the agreed timeframe and satisfying internal and external stakeholder pressures.

Objectivity

Internal conversations about focal areas within the security program are happening daily, with formal discussions taking place during cadenced meetings of an organization’s Cyber Risk Committee or a similar governing body. Questions of priority, status, and ownership are abuzz as stakeholders vie for project precedence, leaving decisions on the table and bottlenecking efficiency.

Through the performance of an external assessment, assessors quell the frustration of office politics, objectively defining a prioritization schema for organizational projects and even identifying previously overlooked or unknown activities that may take rank.

Expertise

The structure and verbiage of typical security frameworks such as the National Institute of Standards and Technology Cybersecurity Framework (NIST CSF), which released an updated version (2.0) in February of 2024, can be convoluted to say the least; it’s like Shakespearian English meets information security.

For most security professionals, understanding the updates made to security frameworks is of little concern. For third-party assessors, however, updates are seamlessly synchronized, allowing them to stay fully current and knowledgeable about the framework(s) in scope.

AI vs. People

As we covered in a previous blog post, AI tools like Large Language Models (LLMs), such as Claude or ChatGPT, have the proclivity to hallucinate and answer fervently, even when incorrect. Let’s look at an illustration using our example of NIST CSF 2.0.

This is a screenshot of a prompt I fed to ChatGPT, requesting a breakdown of Risk Management-related categories from NIST CSF 2.0. You’ll notice the model correctly identified said categories from the newest function of the updated framework, Govern (GV). However, the new and improved framework no longer contains category ID.RM nor ID.GV, both previously present in version 1.1.

Now, if I were not intimately familiar with both versions of NIST CSF, I may not have readily identified this mistake. The upbeat, affirming tone of these LLMs is almost lulling. But here we see the detriment of the siren song, as many a user might be inclined to blindly follow the agent’s guidance. Through external assessments performed by knowledgeable, practiced human beings, incorrectness becomes a non-factor.

How K logix Can Help

K logix’s Cyber Risk Consulting group is comprised of individuals who specialize in a variety of areas including assessment services. From risk assessments to framework assessments to ransomware and threat assessments, our team is equipped with the skills necessary to offer insight into the maturity and orientation of your security program against any security framework. Through our process, we will ascertain the most pertinent areas of improvement to address, alongside feasible recommendations to be implemented for enhancement and continued alignment.