My credential rule reported 842 secrets in vercel/ai. The real count was 0.

The no-hardcoded-credentials rule was designed to identify potential security risks by detecting hardcoded credentials in code. However, when applied to the vercel/ai repository, it reported an alarming 842 findings. Upon further investigation, it was revealed that the vast majority of these findings were false positives.

The primary cause of these false positives was the rule's inability to distinguish between actual credentials and non-credential strings, such as TypeScript union-type literals and error class names. The string 'test' was also frequently misidentified as a credential.

This highlights the limitations of context-blind regex-based detection and the need for more sophisticated, context-aware approaches to identifying security risks.

The development of more accurate detection methods is crucial, especially as AI assistants can inadvertently regenerate the exact strings that fool these detectors, leading to a cycle of false positives and wasted resources.

To improve the accuracy of credential scanners, developers can explore the use of machine learning-based approaches or more advanced regex patterns that take into account the context in which potential credentials are used.

Source: My credential rule reported 842 secrets in vercel/ai. The real count was 0.. Read the full piece at the source.