
Long-context AI models condense conversations and activity histories to continue processing. ARCBench verifies whether prohibitions remain intact within these summaries by performing actual tool operations. Using the pi execution platform, it creates activity histories, then issues rule-violating requests after compression, comparing the results with uncompressed scenarios. It also examines whether rules are preserved, weakened, or erased in the summaries. For example, if a rule prohibits sending emails outside the company, calling a tool to send an email to an external recipient is flagged as a violation. In an evaluation using 17 test cases, the rules themselves were often still present after compression, even in many violations. However, issues also arose on the action side—such as treating subsequent requests as permitted. Simply checking whether rules are written in the summary is insufficient; it is essential to verify that no operations violating those rules are actually executed.

