When AI Writes Your Policies: A Cautionary Tale

Why Policy Libraries Fall Behind

Maintaining a robust policy library is a standard requirement for most widely recognized security frameworks. Standard areas of organizational security programs, such as risk management, vendor management, or change management, must be accompanied by documentation that allows for communication of organizational security expectations, roles and responsibilities, and current state practices.

Whether aligning to a certifiable body like ISO 27001 or simply best practice, security leaders are tasked with updating their libraries at least annually, ensuring they are reflective of current state practices. However, when the time comes for annual reviews, resourcing constraints lead to governance documentation creation and updates falling by the wayside, leaving behind a trail of outdated, unapproved, and inconsistently formatted documentation.

As such, many organizations look to external consultants or, nowadays, AI tools, to delegate the burden of policy library reviews. In an effort to save cost, AI tools are becoming increasingly more popular to perform these tasks, though, be warned; results may vary. So, if you, a security leader, are considering doing the same, let’s talk through a plausible scenario – a cautionary tale, if you will.

A Compliance Deadline Meets an AI Quick Fix

Ahead of an audit, a financial services company identified several documentation gaps against its required framework, including:

• Undefined roles and responsibilities
• Missing documentation for technical controls
• Five policies that did not yet exist
• Fifteen policy documents that no longer reflected organizational practices

Where the AI Approach Broke Down

With the clock ticking by, the company attempted to utilize Claude to ensure their policy and procedure library was up-to-date and in alignment with all necessary legal and regulatory requirements. The organization noted they found it difficult to get Claude up-to-speed on current state practices, as the tool only had access to documentation that was now outdated.

Despite this unforeseen burden, they continued marching forward with Claude at the helm, hopeful the AI would be successful enough in its endeavors to position them well for the audit. Upon review of the large language model’s (LLM) output, however, the organization identified several missing, incorrectly mapped, or outdated controls (i.e., belonging to a previous version). With a looming compliance deadline, the organization recognized the need for swift remediation; that’s where K logix comes in.

How K logix Remediated the Gaps

At the eleventh hour, the firm asked our consultants to rectify Claude’s missteps. The K logix Consulting team began by reviewing existing documentation to understand the customer’s needs and gaps. Through interviews with relevant stakeholders, the team was able to glean a thorough understanding of current practices, as well as outstanding areas for alignment. Utilizing information from the interview sessions in tandem with existing documentation, K logix updated the company’s documentation to be reflective of their current state, applying mappings to necessary requirements. With K logix’s performance of this exercise, the organization was able to remediate the gaps identified by K logix consultants to further bolster compliance efforts and ultimately pass their audit.

The Takeaway: AI Still Needs Expert Oversight

AI is purporting to be the new cloud, with organizations worldwide seeking opportunities for implementation across the enterprise. However, it is still new, making mistakes as any entry level analyst might. As the kinks are ironed out (though, keep in mind, there will always be a few wrinkles), it’s important to temper expectations as it pertains to AI use, recognizing that, while it is useful in many aspects, AI requires more handholding and oversight than might be expected. For certain use cases, AI serves as a reliable quick fix, whereas for others, such as the creation of a well-tailored policy library, fine tuning is required.