← Back to Jason Harvey’s portfolio
Project overview
This project began as an experiment using the free DeepSeek coding model to build a Python security scanner. The first version provided a useful starting point, but early validation exposed weaknesses in credential detection and report handling. I then used ChatGPT to take over the independent code review, make corrections, expand regression tests and document the limitations.
The aim is to explore how AI-generated code can be tested and improved through repeatable security-focused validation, rather than assuming generated code is correct.
What the scanner checks
Credential exposure
Patterns for AWS access keys, GitHub tokens, API keys and generic hardcoded secrets.
SQL injection patterns
Risky query construction using concatenation, f-strings and formatting patterns.
Web configuration
Missing security headers and insecure HTTP references.
Unsafe code and settings
Checks involving eval, exec, unsafe pickle use, debug mode and hardcoded passwords.
Reports
HTML findings with severity levels and remediation guidance, with secret masking and duplicate-finding reduction.
Test targets
Individual files and directories, including a deliberately vulnerable synthetic sample application.
Testing and improvements
- Early credential testing found that the initial scanner detected 8 of 13 deliberately planted fictional credential assignments. It missed some quoted dictionary-key and AWS-related formats.
- Review also identified problems with unmasked secret values, duplicate findings and inconsistent single-file scanning.
- The follow-up work prioritised provider-specific credential patterns, corrected the synthetic AWS test expectation, improved secret masking and deduplication, and cleaned generated files from the repository.
- Verified regression result on 2 November 2026: 98 tests passed, 0 failed.
What this experiment does not prove
- It does not actively probe a public website or test a running service over the network.
- It is not a substitute for dynamic application security testing, dependency scanning, a professional penetration test or a formal security audit.
- Pattern-based checks can miss unfamiliar credential formats and can produce false positives. Every finding needs human review.
- The sample application contains fictional vulnerable examples for controlled testing and must not be deployed.
A future development step would be to improve credential-format coverage, audit every report output for accidental secret leakage, add CI and broaden the synthetic test corpus.
Technology and project links
Python · pytest · static analysis · regular-expression detection · HTML reporting · AI-assisted development and code review.
DeepSeek is credited for the initial implementation. ChatGPT supported the subsequent independent review, fixes, regression testing and documentation. Claude was discussed as a possible independent reviewer, but was not used to produce the recorded test results.