# Preparation
In this section we'll discuss in detail how to prepare your organization to
launch and run a successful vulnerability disclosure program, including
practical advice on how to fill in the gaps you've identified.
## Finding bugs
Augmenting your existing security program with outside security researchers
is a great way to find complex issues and obscure vulnerabilities. However,
using a VDP to find basic security problems that could be discovered internally
is a waste of resources.
## Asset management
When it comes to finding bugs, the only way to know where to start is if you
have a good idea of what's out there. You can buy a hundred security tools, but
it won't make any difference if teams are standing up applications, systems, and
services ad hoc without your knowledge, especially if you don't have a way to
discover and perform security assessments against these assets. Check with the
individuals and teams that are responsible for helping stand up new
applications, systems, and services to see if they have a process in place for
creating and maintaining an inventory of what's getting spun up and who owns it.
If there isn't a current process, this is a great opportunity to collaborate
with these teams to build one. Gaining an understanding of your organization's
assets is the best place to start when identifying your attack surface. As part
of this process, the security team should be involved in the development of new
infrastructure implementation to provide [security reviews](https://developers.google.com/android/play-protect/starting-a-vdp/docs/preparation#review-process). It is good practice to
have an extensive inventory of assets and owners. This kind of inventory is
useful when applying new patches that require certain systems to be temporarily
shut down. It provides a road map of individuals or teams that need to be
informed and which systems are affected. Having a robust asset management
process in place ensures owners are identified earlier in the process, updated
regularly, and that all systems across the organization operate as
intended.
In addition to proactive asset management, consider what reactive measures
you can implement as well to identify assets that belong to your organization,
but slipped through the cracks of your standardized asset management processes.
This can include using the same "reconnaissance" processes used by security
researchers that participate in VDPs and . For example, you
can leverage free and open source tools that scan and enumerate Internet-facing
IP ranges or domains that may belong to your organization. A Google search for
[bug
bounty recon](https://www.google.com/search?q=bug+bounty+recon) will produce a variety of tips and tricks to help you identify
assets from your organization you were unaware of.
## Basic vulnerability scanning
Now that you have a solid foundation of *where* you need to find
security issues, let's dive into *how* you actually do that. There are
various levels of depth that you can go into depending on the resources of your
organization but you need to find a balance between your internal security
efforts and the external hacking community through your vulnerability disclosure
program. This balance is different for every organization, depending on the
resources available.
### Choose your tools
There are many different tools to help with identifying vulnerabilities. Some
vulnerability scanning tools are available for free, while others come at a
cost. Figuring out what tools to pick depends on your individual needs.
- Your organization's requirements
- How well each tool satisfies these requirements
- If the benefits of the tool outweigh the costs (financial and implementation).
You can use this **requirements template**
([OpenDocument .ods](https://developers.google.com/static/android/play-protect/starting-a-vdp/assets/requirements-template.ods),
[Microsoft Excel .xlsx](https://developers.google.com/static/android/play-protect/starting-a-vdp/assets/requirements-template.ods))
to evaluate various tools against your requirements.
Some example requirements are included in the template, but you should discuss
with your security, IT, and engineering teams to align on required capabilities.
Before launching a vulnerability disclosure program, at a minimum, you'll want
to be able to perform vulnerability scans against any externally facing assets
(such as websites, APIs, mobile apps). This will help you find and fix easily
discoverable vulnerabilities before you invite external security researchers to
test against these assets and services.
### Scanning setup
Automated vulnerability scans can find a lot of issues, but they can also
produce false positives. That is why it is necessary to have resources to
validate the results before sharing them with the impacted teams. You'll need to
implement processes to ensure scans are run on a regular basis, and that the
results of these scans are actually addressed. This will look different for
every organization, but at a minimum, you'll want to determine:
- Scan frequency
- Continuous
- Weekly
- Monthly
- Which assets are being scanned
- Credentials
- Authenticated vs. unauthenticated scans
- (hint: if you do *not* scan with credentials, then a security researcher tests with credentials when you start your VDP, you might get a large spike of identified vulnerabilities)
- Roles and responsibilities
- Identify the team members responsible for running the scans
- Set up a rotation if necessary
- Scan results
- Verifying scan results
- Filing bugs for verified vulnerabilities
- Identifying owners to fix bugs
- Following up with owners on remediation
We'll go into more detail how to ensure identified security issues are fixed
in the [Fixing Bugs](https://developers.google.com/android/play-protect/starting-a-vdp/docs/preparation#fixing-bugs) section later in
this guide.
## Security review process
While vulnerability scanning is a great way to reactively identify security
issues in your organization, implementing security review processes can help
prevent vulnerabilities from being introduced in the first place. For the
purpose of this guide, the term **security review** refers to any
situation that triggers a manual review by a member of your security team.
Typically, this includes having the authority to block a change if it's deemed
too risky. If your security team doesn't have the ability to block risky
changes, you'll still want to have processes in place to document the risk. This
can help ensure whoever is pushing for the change knows the risk involved, and
proactively accepts that risk.
### Security review criteria
When should security reviews happen? Creating a set of criteria that triggers
a security review helps ensure everyone is on the same page. Below are some
examples of scenarios that might trigger a security review.
- New functionality related to sensitive user data is proposed
- A new feature that allows users to share their location on map
- Requesting potentially sensitive information from users, such as their home address, date of birth, or phone number
- Major updates to existing functionality is made
- Taking existing user data and using it in a new way that users might not expect without giving them an opportunity to opt out
- Changes to any features related to authentication, authorization, and session management
- Changes to the company's production environment
- Network configuration changes, especially changes that might result in exposing services externally
- Installation of new software that handles sensitive user data, that if compromised could indirectly be used to access sensitive user data
- Standing up new systems or services
- Interacting with a new vendor or changing how you work with an existing vendor
- Onboarding a new vendor that will handle sensitive user data
- Changes to how you work with an existing vendor that results in the vendor handling sensitive user data
This is not an exhaustive list, but it should get you thinking about what
sorts of changes should require a security review. As you define the criteria of
what does and doesn't require a security review, talk it over with key
stakeholders across the organization to ensure:
- Stakeholders have a chance to review and provide feedback on the criteria
- Stakeholders agree to the criteria
- Stakeholders agree to proactively request security reviews
Document this criteria, as well as how to request a security review (for
example, filing a bug to a queue the security team monitors) to make it as easy
as possible for your organization to follow this process.
### Security review resourcing
Unlike automated scans, security reviews can be more resource intensive to
perform. Every security team only has so much time in the day to accomplish a
myriad of tasks, so you'll need to estimate how many security reviews may be
generated based on your criteria. If you find your team is overwhelmed and
falling behind, those waiting for their features to launch will become upset
with the security team. This can cause a cultural shift in the organization
causing the security team to be viewed as a blocker instead of a partner. If the
security review process isn't efficient, many individuals and teams will try to
bypass it completely. If resources are tight, consider loosening your criteria
for requiring a security review, and be willing to accept some more residual
risk. If incidents do occur as a result of lack of resources to perform security
reviews, this will help justify the need for more security resources.
### Performing security reviews
When it comes to deciding on which security reviews to perform and how to
perform them, you'll need a prioritized queue to pull from. Create a
standardized way for others in your organization to request a security review
with whatever information you'll require to prioritize it appropriately. For
example, consider a questionnaire that includes items such as the nature of the
change, including a brief summary of the change and what types of user data may
be impacted. You can automatically categorize potential security reviews into
high, medium, or low risk changes based on the answers to these questions. If a
change is high risk, you may require a more in depth security review process. If
a change is lower risk, you may be able to implement a more lightweight security
review process to help reduce resources required and speed up the process,
better enabling the business. Consider setting up a rotation within your team to
be responsible for managing the security review queue, ensuring that new
security reviews are picked up by members of your team, and following up on
progress of existing security reviews. The actual process of the security review
will vary depending on what's being examined. For example, a new feature in your
mobile app might require a security engineer to review the code and look for
potential vulnerabilities. New software being installed might need to be
reviewed to ensure access control is set up appropriately. Working with outside
vendors can present an entirely different process. For reference, read through
Google's [Vendor
Security Assessment Questionnaire](https://github.com/google/vsaq).
## Fixing bugs
Finding bugs is important, but security only improves after those bugs are
fixed. Knowing what risks exist to your organization is good, but being able to
efficiently address that risk is better.
## Vulnerability management
Vulnerabilities come from a variety of resources, including internal efforts
(for example, vulnerability scans and security reviews), third party penetration
tests and audits, or even external security researchers that notify you through
support channels before your VDP is officially launched. Your organization needs
a way to categorize new and existing vulnerabilities to ensure they are
communicated to the right stakeholders, prioritized correctly, and fixed in a
timely manner. When you launch your VDP, you'll have a new stream of
vulnerabilities entering your vulnerability management processes. Having solid
processes in place for handling these vulnerabilities helps you track progress
towards remediation and respond to requests from external security researchers
for updates. Being able to quickly prioritize a vulnerability and communicate
with VDP participants about remediation timeline will increase engagement with
the security researcher community, as well as improve the reputation of your
organization's security. The following sections outline various aspects of your
vulnerability management program you'll want to have in place before launching
your VDP.
### Establish severity standards and remediation timelines
Creating a common language around the severity of vulnerabilities and ideal
remediation timelines associated with each severity makes it easier to set
standard expectations with your organization. If every vulnerability is treated
like an emergency, your organization will exhaust their resources and grow
resentful towards the security team. If every vulnerability is considered low
priority, vulnerabilities will never get fixed, and the risk of a breach
increases. Every organization has limited resources, so you'll need to establish
a severity ranking. This ranking provides criteria that helps your organization
understand what severity a vulnerability falls into, and expected remediation
timelines associated with each severity. Draft a set of severity guidelines and
share it with key stakeholders in your organization for feedback. For example,
if engineering is involved in crafting your severity standards, they'll more
likely buy-in to these standards and adhere to them when it comes time to fix a
vulnerability within a specified timeframe. These severity guidelines may vary
depending on what risks are specific to your business. You may want to consider
a threat modeling exercise to think about what threats are most likely and
impactful to your organization, and include examples of issues that would fall
into each severity category. Below is an example of severity standards and
remediation timelines for a financial organization.
| Severity | Description | Remediation Timeline | Example(s) |
|---|---|---|---|
| Critical | Issues that pose an imminent threat to our users or our business. | **Owner:** A primary owner for ensuring the vulnerability is fixed should be identified within 8 hours. Call and page resources as needed, even outside of normal business hours. **Fix:** The issue itself should be fixed, or at least have risk mitigated, as soon as possible, or at most, within three business days. | Compromise of a production database including all users' financial records.
An attacker gaining access to trade secrets, such as our proprietary investment algorithms.
An active incident including an attacker gaining access to our internal network or sensitive production systems. |
| High | Issues that, if exploited, could cause significant damage. | **Owner:** A primary owner should be identified within one business day. **Fix:** Within 10 business days (\~2 weeks). | Vulnerabilities that could result in access to sensitive user data or functionality (e.g. ability for any user to steal funds from another user). |
| Medium | Issues that are harder to exploit or do not result in direct damage. | **Owner:** A primary owner should be identified within five business days. **Fix:** Within 20-40 business days (\~1-2 months). | Verified issues identified by automated scanners, such as patches for security updates without known exploits.
Information disclosure issues that would likely help with further attacks.
Rate limiting issues that could potentially be exploited (e.g. being able to continuously guess passwords for a user). |
| Low | Issues with minimal impact; primarily used for logging known issues. | No requirements for finding an owner or fixing within a specified timeline. | Information disclosure that does not present likely risk, but where the information does not need to be externally accessible. |
### Grooming bugs
We're not talking about haircuts here, we're talking about ensuring bugs are
formatted correctly so they can be easily fixed. Using the previous table as a
guideline, establish your own severity definitions. These definitions are used
to classify bugs into various severities and communicate them to owners.
On top of assigning each vulnerability a severity, you'll need to ensure your
bugs are in a standard format that makes it easier for receiving teams to
process. Vulnerabilities will enter your vulnerability management processes in a
variety of formats (such as automated scanner results or manual write-ups from
security reviews). Taking the time to convert each vulnerability into a standard
format will increase the chances of the receiving team being able to quickly
understand and address the issue.
This format or template might vary depending on your organization and what
information is most pertinent to help owners fix bugs assigned to them, but
here's an example template you can use. You'll be able to reuse this template
later on when you create your vulnerability disclosure program submission form
for researchers.
**Title:** \ **Summary:** \ **Reproduction Steps:** \ **Impact / Attack Scenario:** \ **Remediation Steps:** \
Here is an example of potential high severity vulnerability:
**Title:** \[HIGH\] Insecure Direct Object Reference
(IDOR) in profile pages
**Summary:** An IDOR was discovered in our app's
profile pages functionality that would allow any user to gain unauthorized
access to view and edit another user's profile, including the other user's full
name, home address, phone number, and date of birth. We've reviewed the logs and
this issue does not seem to have been exploited yet. This issue was discovered
internally.
**Reproduction steps:**
1. Set up a proxy for example, Burp Suite) to intercept traffic on a mobile device with the app installed.
2. Visit your profile page and intercept the associated HTTP request.
3. Modify **profileID=######** to be **profileID=000000** (this is a test user) and send along the HTTP request.
4. The app will show the profile of user 000000, and you will be able to view and edit their information.
**Attack scenario / impact:** Any user can use this
vulnerability to view and edit another user's profile. In the worst case
scenario, an attacker could automate the process of retrieving every user's
profile info in our entire system. While we don't believe this has been
exploited yet, it's important that we treat this as a standard HIGH severity
issue. If we observe evidence of exploitation, this could escalate to CRITICAL
severity.
**Remediation steps:** Implement server-side checks to
ensure the user making the request should have access to view/edit the profile
requested via the value of profileID. For example, if Alice is logged in and has
profileID 123456, but Alice is observed to have requested profileID 333444, the
user should see an error and this attempt to access another user's profile
should be logged. For more information on IDOR and how to fix it, please see
[OWASP's
materials on this bug](https://www.google.com/search?q=owasp+idor).
You can save time and manual effort by finding ways to automate converting
vulnerabilities from various sources into your standard format. As you create
more vulnerabilities, you might find common themes in remediation steps. Beyond
your generic bug format template, you might want to create additional templates
for common vulnerability types.
### Finding owners
Perhaps one of the most difficult aspects of vulnerability management is
identifying owners to help fix bugs, as well as getting their buy-in to dedicate
resources towards actually fixing bugs on schedule. If you've set up [asset management](https://developers.google.com/android/play-protect/starting-a-vdp/docs/preparation#asset-management) processes, this will be
a bit easier. If not, this might serve as motivation to do so. Depending on the
size of your organization, finding an owner might be fairly simple, or
incredibly complex. As your organization grows, the effort to determine who is
responsible for fixing newly discovered security issues also grows. Consider
implementing an operational on-duty rotation. Whoever is on-duty is responsible
for reviewing unassigned vulnerabilities, tracking down owners, and prioritizing
based on severity. Even if you're able to identify who is responsible for fixing
a vulnerability and assign them to the bug, you'll also need to persuade them to
invest time in actually fixing it. This approach might vary based on the team or
individual, and what other items they are working on. If you've achieved
organizational buy-in on your [severity
standards and remediation timelines](https://developers.google.com/android/play-protect/starting-a-vdp/docs/preparation#severity-standards), you can refer to those, but
sometimes it may take extra persuasion to get someone to fix a bug. Here are
some general tips for driving remediation of vulnerabilities:
- **Explain why** :When someone is assigned a vulnerability to fix, it's usually unexpected work. Explain why this issue is important to fix in a timely manner (e.g. the Impact / Attack Scenario) and ensure the owner understands.
- **Gather context**: In some cases, only one person has the knowledge necessary to fix a bug, and they might have other tasks they're working on. Take the time to find out what these are - it's possible that the other tasks may be more important than fixing this vulnerability in the near term. Demonstrating empathy and flexibility on remediation timelines will help earn goodwill and strengthen your relationship with those you need to fix vulnerabilities. Just be careful not to give too much leeway, otherwise your organization won't take your remediation timelines seriously.
- **Explain how:** Even if you include remediation advice in the bug, the owner of fixing the issue might be confused or need help learning how to fix the bug. If they need help figuring out how to fix it, help teach them. Simply throwing bugs at owners without helping them will hurt the organization's relationship with the security team. Helping others as much as possible will empower them to fix present and future vulnerabilities, as well as to help teach others.
- **Adapt your request** :Various teams and individuals may have existing processes for how they accept and prioritize incoming work requests. Some teams may want all incoming requests to come through their managers. Others will want requests for help to be submitted in a standard format. Some will only work on what's been predefined in a sprint. Whatever the case, taking some extra time to adapt your request to fit the format the team or individual usually uses to intake requests for help will increase the likelihood of your request being prioritized and actioned.
- **Escalate as a last resort**: If you've tried all of the above, and the individual or team responsible for fixing a vulnerability just won't take the time to fix a serious security issue, consider escalating to leadership as needed. This should always be a last resort, as it can damage your relationship with the individual or team in question.
## Root cause analysis
In addition to finding and fixing individual vulnerabilities, performing root
cause analysis (RCA) can help you identify and address systemic security issues.
Everyone has limited resources, so it's tempting to skip this step. However,
investing time into analyzing trends in your vulnerability data, as well as to
look further into critical and high severity bugs, can save time and reduce risk
in the long term. As an example, let's say you notice the same vulnerability
type (for example, [intent
redirection](https://support.google.com/faqs/answer/9267555)) appear over and over again throughout your app. You decide
to talk to the teams that are introducing this vulnerability into your app, and
realize the large majority of them do not understand what intent redirection is,
why it matters, or how to prevent it. You put together a talk and a guide to
help educate developers in your organization about this vulnerability. This
vulnerability probably will not completely disappear, but the rate at which it
appears will likely decrease. When you launch your VDP, every vulnerability
reported to you by a third party is something that slipped through your existing
internal security processes. Performing RCA on bugs from your VDP will provide
even further insight into how to systematically improve your security
program.
## Detection and response
Detection and response refers to any tooling and processes you have in place
to detect and respond to potential attacks against your organization. This could
come in the form of either purchased or self-developed solutions that analyze
data to identify suspicious behavior. As an example, in the [Grooming Bugs](https://developers.google.com/android/play-protect/starting-a-vdp/docs/preparation#grooming-bugs) section we talked about
logging every time a user attempts to gain unauthorized access to another user's
profile. You might have a signal or alert that's generated if you notice a user
generating a large number of failed attempts to access other user's profiles in
a short period of time. You might even automate the process of blocking that
user from accessing any of your services for a certain period, or indefinitely
until the situation can be reviewed and restore access manually. If you don't
already have detection and response mechanisms in place, consider working with
an expert consultant to help guide you on how to build a digital forensics and
incident response (DFIR) program for your organization. If you do have detection
and response mechanisms already in place, you'll want to consider the
consequences of having five, ten, or even one hundred security researchers
testing against your Internet-facing attack surfaces. This can have a big impact
on any IDS/IPS (intrusion detection and prevention systems) you have in place.
Potential risks include:
- **Alerts overload:** A flood of alerts or signals that look like malicious attacks, but are actually normal, approved testing from security researchers participating in your VDP. So much noise can be generated that it becomes difficult to distinguish real attacks from legitimate security testing.
- **Incident response false alarms:** If you have processes in place that page individuals at 2:00AM on Saturday, they will not be happy about waking up and investigating a potential breach that was actually just a security researcher performing legitimate testing.
- **Blocking security researchers:** If you have aggressive IPS (intrusion prevention systems) in place, you might end up blocking the IP address of a security researcher that's attempting to run scans, manual tests, etc. to identify vulnerabilities and report them to you. Especially in the case of a VDP, if a security researcher gets blocked after five minutes of testing, they may lose interest and instead focus on another organization's program. This can result in an overall lack of engagement in your program from security researchers, which increases the risk of vulnerabilities remaining undiscovered (and thus unknown to your organization). While you may not want to tone down your IPS itself, there are other measures you can take to mitigate the risk of disengaging researchers.
How to address these risks depends largely on what approach you want to take
towards working with external security researchers. If you want a more
**black box** style of testing that simulates real attacks, then
you might do nothing. In this case, researcher's traffic will generate alerts
and your team may take actions to investigate and respond accordingly. This will
help your team get practice in responding to what looks like real attacks, but
might decrease engagement with security researchers, especially if they are
blocked from testing. It may also result in missing a real attack while spending
time investigating legitimate testing. If you want more of a **gray
box** approach, you can consider working with security researchers to
self-identify their testing traffic in some way. This will enable you to
whitelist or otherwise filter out traffic from their testing and resulting
alerts. Your team will be able to better distinguish real attacks from approved
testing, and researchers will be empowered to find and report vulnerabilities to
you without being hindered by your intrusion prevention systems. Some
organizations ask security researchers to submit a form to apply for a unique
identifier that can be attached to headers in requests generated by the
researcher. This enables the organization to whitelist traffic for the
researcher, as well as the ability to identify if the researcher starts to go
beyond the agreed of testing.
If this happens, you can reach out to the researcher and ask them to hold
off until you can work together on a testing plan.