How to Run a 30-Day SAST and SCA Pilot

A 30-day SAST and SCA pilot fits into four stages: a week for preparation and installation, two weeks for scanning and reviewing results, and a week for conclusions and the decision. The key is to fix the goals, the set of repositories and measurable success criteria in advance. Otherwise, instead of a product comparison, you will end up with a collection of impressions.

Why run a pilot, and how it differs from a demo

A vendor demonstration shows the product’s capabilities on a prepared example. A pilot (proof of concept) shows how the product works on your code, in your pipeline and with your people. It answers questions a presentation cannot: how many real vulnerabilities it will find, how much noise it will create, how long scanning takes, and whether developers will agree to work with it. The selection criteria to start from are described in the article “How to Choose a SAST Tool”.

Week 0: preparation before the start

This work determines the success of the whole pilot, so do it before installing any products.

• Formulate the goals. For example: choose SAST and SCA for all teams; confirm operation in a closed network; assess the impact on build time.
• Assign roles. A pilot owner from the security side, a technical lead from DevOps, 2–3 participating developers, and a representative of the vendor or integrator.
• Choose repositories. 3–5 projects: different languages and teams, one project with known vulnerabilities from past penetration tests, and one project with a long dependency history. This gives you a benchmark for checking misses.
• Agree on access. Git accounts, CI permissions, network rules, and a place to install the scanning server. Delays with access are the most common reason a pilot misses its deadlines.
• Define the metrics. See the section below.
• Settle the data question. If code cannot leave your perimeter, request only a local installation and confirm this in writing.

Week 1: installation and first run

• Install the solutions in a test environment and measure the effort: how many hours and which specialists were needed. This goes into the total cost of ownership estimate.
• Connect the repositories and run the first scan with default settings. Record the number of findings and the scan time.
• For SCA, check how the product determines dependencies: from manifests or from the build, and whether transitive dependencies are included. For example, Sonatype Lifecycle supports more than 20 languages and package formats and generates SBOMs, while OWASP Dependency-Track accepts SBOMs in CycloneDX format. So clarify which input format each product needs. For more on SBOMs, see “What Is an SBOM”.

Weeks 2–3: reviewing results and CI integration

• Triage together with developers. Take a random sample of findings (for example, 50–100 per product) and mark each one as true, false or “not relevant”. If you are comparing several candidates, triage blind, without knowing which product produced each finding.
• Check for misses. On the project with known vulnerabilities, verify whether each tool found what a pentest has already confirmed.
• CI/CD integration. Connect pull request scanning in GitLab, GitHub or Jenkins. Measure the check time and confirm that the incremental mode works. OWASP recommends running SAST precisely on pull requests.
• SCA policies. Configure a rule that stops the build on critical vulnerabilities and prohibited licences, and see how many builds it would have stopped. Enable warning mode first so you do not paralyse the teams’ work.
• Protection against malicious packages. If you are evaluating this capability, test it separately. At Sonatype it is the job of the Firewall product, which blocks malicious components at download and quarantines suspicious ones.
• Configuration. Try suppressing false positives, adding a custom rule, and assigning tasks in Jira. Assess how easy this is for your team.

Week 4: results and decision

• Collect the metrics into a single table for each candidate.
• Hold a retrospective with developers: what got in the way, what helped, and whether they are ready to use the tool every day.
• Request a commercial offer with a three-year calculation and add the cost of infrastructure and maintenance. The calculation approach is described in “Open-Source or Commercial AppSec Tools”.
• Prepare a short report for management: what was tested, what was found, which solution you recommend, and which risks remain.

Which metrics to collect

• Share of true positives: true findings in the sample divided by the sample size.
• Coverage of known vulnerabilities: how many of the previously confirmed defects were found.
• Scan time: for a full repository and for a pull request.
• Language and dependency coverage: whether all technologies in the stack are analysed, with no “blind spots”.
• Implementation and maintenance effort: in person-hours.
• Time to triage one finding: measured with developers.
• Developer convenience: a short survey of participants after the pilot.
• Operation in your environment: whether installation without access to the external network has been confirmed, if that is a requirement.

Agree on threshold values in advance, for example “the share of true positives is not below an agreed figure” and “pull request checks take no longer than a set time”. The specific values depend on your organisation; there are no universal standards.

Common pilot mistakes

• No written success criteria. In the end, whoever argues more convincingly decides.
• Piloting only on demo projects. They do not reveal real noise or complexity.
• Developers not involved. Without them it is impossible to assess true and false positives.
• Too many candidates. Two or three products is the limit at which each one can be evaluated properly within 30 days.
• Different conditions for candidates. Use the same repositories, the same depth of configuration and the same deadlines.
• A pilot without a decision. Define in advance who decides and when, otherwise the project drags on.

What comes after the pilot

After choosing a product, expand coverage in stages: first the critical applications, then the other teams. Start in warning-only mode, then introduce blocking for critical findings. For the full picture, consider a DevSecOps map: DevSecOps implementation services help embed the tools into your processes rather than just install them. Product comparisons for the pilot: SAST and SCA.

Pilot report template

So that the pilot results can be presented to management and used in procurement, write them up as a short five-section document. The first section covers goals and scope: which repositories and products took part, and under what conditions. The second is a table of metrics for each candidate. The third is developer feedback: what was convenient and what got in the way. The fourth is an estimate of effort and total cost of ownership over three years. The fifth is the recommendation and an implementation plan with stages and owners.

Attach examples of real findings to the report: two or three confirmed vulnerabilities the tool found, and two or three typical false positives. Such examples persuade management better than any percentages, because they show both the value and the costs on concrete cases from your code.

Remember the organisational side too. Warn the development teams in advance that additional checks will appear in their pipelines during the month, and explain that the pilot results will not be used to evaluate individual developers. Without this, participants start hiding problems or bypassing the scans, and you get a distorted picture. Set up a single channel for pilot questions and a short weekly meeting to discuss blockers and interim figures.

If the pilot runs in several countries or across several legal entities of a group, define a single set of metrics in advance. That way the results can be compared across sites, and a joint purchasing decision and standard can be made for all teams.

Frequently asked questions

Sources

• OWASP DevSecOps Guideline: Static Application Security Testing
• Sonatype Lifecycle: product page
• Sonatype Firewall: product page
• OWASP Dependency-Track
• Semgrep Docs: Semgrep Code overview