In a digital age dominated by the rapid evolution of artificial intelligence led by ChatGPT, the recent spike in ChatGPT jailbreak attempts has sparked a crucial discourse on the sustainability of AI systems and the unintended consequences these breakthroughs pose for cybersecurity and the ethical use of AI. A research paper “AttackEval: How to Evaluate the Performance of Jailbreak Attacking on Large Language Patterns” was recently released. introduces a new approach to evaluate the effectiveness of jailbreak attacks on large language models (LLMs) such as GPT-4 and LLaMa2. This study departs from traditional resilience-focused evaluations by proposing two different frameworks: a coarse evaluation and a fine evaluation, each using a point range from 0 to 1. These frameworks allow for a more comprehensive and nuanced assessment of attack effectiveness. Additionally, the research has developed a comprehensive ground-truth dataset specifically tailored for jailbreaking tasks, serving as a benchmark for current and future research in this evolving field.
The study addresses the growing urgency in evaluating the effectiveness of prompts to attack LLMs due to the increasing sophistication of such attacks, particularly those that force LLMs to generate prohibited content. Historically, research has focused primarily on the resilience of LLMs, often neglecting the effectiveness of attack prompts. Previous studies that focused on performance often relied on binary metrics, categorizing outcomes as successful or unsuccessful based on the presence or absence of illegal outcomes. This study aims to fill this gap by introducing more sophisticated assessment methodologies, including both coarse and fine-grained assessments. The coarse-grained framework evaluates the overall effectiveness of prompts in various baseline models, while the fine-grained framework delves into the intricacies of each attack prompt and the corresponding LLM responses.
The research has developed a comprehensive jailbreak truth data set that is carefully curated to cover a diverse set of attack scenarios and rapid variations. This data set serves as a critical benchmark, allowing researchers and practitioners to systematically compare and contrast the responses generated by different LLMs under simulated jailbreak conditions.
The key contribution of the study includes the development of two innovative evaluation frameworks for evaluating attack prompts in jailbreak tasks: a coarse evaluation matrix and a fine evaluation matrix. These frameworks shift the focus from the traditional LLM emphasis on resilience to a more focused analysis of the effectiveness of attack prompts. The frameworks introduce a nuanced scaling system ranging from 0 to 1 to precisely measure gradations of attack strategies.
The vulnerability of LLMs to malicious attacks has become a growing concern as these models become increasingly integrated across sectors. The study examines the evolution of LLMs and their vulnerability, particularly to sophisticated attack strategies such as fast injection and jailbreak, which involve fine-tuning or tricking the model into producing unwanted reactions.
The research evaluation method includes two different criteria: coarse-grained and fine-grained evaluation matrices. Each matrix generates a score for the user’s attack prompt, reflecting the effectiveness of the attack prompt when manipulating or using the LLM. An attack prompt consists of two key components: the context prompt and the dangerous attack question.
For each attack attempt, the study entered the attack prompt into a series of LLMs to obtain an overall performance score. This was done using a selection of prominent models, including GPT-3.5-Turbo, GPT-4, LLaMa2-13B, vicuna, and ChatGLM, with GPT-4 as the judgment model for evaluation. The study meticulously calculated a separate resilience weight for each model, which was integrally applied throughout the scoring process to accurately reflect the effectiveness of each attacking prompt.
The survey evaluation approach includes four main categories for evaluating LLM responses: full refusal, partial refusal, partial compliance, and full compliance. These categories correspond to the corresponding scores of 0.0, 0.33, 0.66, and 1. The methodology uses conventional methods to determine whether a response contains illegal information and then categorizes the response accordingly.
The study used three evaluation matrices: coarse-grained, fine-grained with ground truth, and fine-grained without ground truth. The dataset used for evaluation was the jailbreak_llms dataset, which included 666 prompts collected from various sources and covered 390 malicious questions focused on 13 critical scenarios.
In summary, the research represents a significant advance in the field of LLM security analysis by introducing new multi-faceted approaches for evaluating the effectiveness of attack prompts. The methodologies offer unique insights to comprehensively evaluate attack prompts from various perspectives. The creation of a ground-truth dataset marks a major contribution to ongoing research efforts and highlights the reliability of the study’s evaluation methods.
To visually represent the complex evaluation process described in the paper, I have created a detailed diagram that illustrates the various components and methodologies used in the study. The chart includes sections for coarse evaluation, fine-grained evaluation with ground truth, and fine-grained evaluation without ground truth, along with flowcharts and graphs demonstrating how attack prompts are evaluated in different LLMs.
Image source: Shutterstock