How to avoid, identify, and quickly sort through AI generated contributor proposals

As AI becomes more common in the open source ecosystem, we’ve seen an increase in low quality contributor proposals. Below are some ideas from your fellow mentors and org admins on how they were able to quickly identify, avoid, and sort through spam proposals.

1. Enforce Pre-Submission Community Interaction

Community interaction is one of the strongest signals of a good proposal. We strongly recommend implementing this into your selection process.

  • Mandatory Chat: Require applicants to join the organization's communication channel (e.g., Slack, Zulip, IRC) and filter out all proposals from candidates who are not on that channel.

    How to track interactions

  • Keep the focus on individuals - not the proposals. Consider starting a shared spreadsheet to keep track of individual prospective contributors and their journey through the community, their strengths and weaknesses, red flags and green flags, relevant mentors, etc.

2. Mandatory Technical & Contribution Requirements

Below are some ideas on how to filter out candidates who haven't put in the basic effort to engage with the codebase.

  • Mandatory PR/Code Challenge Link:
    • Ask for a link to 1-3 most meaningful PR / contributions they did to the project
    • Require minor fixes, code challenges, GitHub discussions/pull-requests

  • Require a Draft Proposal: Ask candidates to share a draft GSoC proposal before the formal submission. (NOTE: with this other candidates may try and copy the original person’s work so be sure to pay attention to creation dates, first edited, etc)

    How to track your org’s requirements

  • Require pre-requisites from all applicants: Requiring specific actions before a formal application is submitted acts as a natural filter, ensuring only committed and capable applicants move forward.

  • What pre-reqs should we require? Often a combination of things: they must communicate with org before submitting proposal, they must do 2-3 PRs, they need to do a test, and/or they have to use a particular template for their proposal. Your org should determine your pre-reqs and make them very clear. The more things you require, the easier it will be to weed out spammy proposals not meeting all requirements.

  • New Pre-reqs button For Org Admins to use and viewable by mentors for each proposal (also a field in the proposal export and a filter on proposals dashboard).

  • Check on Contributor GitHub Profile/Activity: The GSoC webapp has a field for the applicant to include their GitHub username and/or a direct link to their contributions with the organization, making it easy to filter proposals where no prior interaction exists.

3. How to Screen for AI Generated Proposals

It’s inevitable! There will be some AI generated submissions. Screen them out quickly with the tips below.

  • Templated Proposal Enforcement: Require the use of a standard template for all proposals to your org. This could simply mean requiring them to answer 4 or 5 specific questions. Enforce automatic disqualification for missing requirements. This is especially helpful for orgs receiving 50+ proposals.
  • Look for Generic Wording: Use a quick search to filter out proposals containing obvious nonsense like:
    • [placeholders]
    • "signal words" like 'revolutionize'
    • Typos are good - they are human!
  • Require applicants to include a 1-minute short video introducing themselves and explaining why they want to work on this project (face must be visible to avoid spam/multiple entries).
  • Most of the AI generated spam just rephrases and expands on the idea description your org listed on the Ideas list. It becomes easier to notice the patterns (e.g. the third paragraph contains "1-3 minutes" because the third sentence in the idea page mentions "1-3 minutes")
  • Super pretty proposals are more likely to be AI-generated. Look at the content not just if there are interesting graphs and pictures (AI can generate these with ease).

4. Optimize your internal process

Make the review process a little less painful for your organization; consider following the tips below.

  • Org-Specific Application Questions: Determine 3-5 questions you want every applicant to answer in their proposal. Make sure this is clearly communicated in the instructions, and use failure to submit this as an automatic rejection filter.
  • Pre-Selection/Initial Admin Assessment: Have an Org Admin do a quick, initial screening to categorize proposals (e.g., Valid, Invalid/Spam, Out of Scope) before distributing them to mentors for in-depth review.
  • Flag Title/Project Mismatches: In instructions require applicants to use specific title or key words to make it easier for OA/mentors to easily match it with the project idea as many mentors are only interested in reviewing the proposals for the project idea they wish to mentor.
    • Require proposals include a standardized header, possibly including a standard format for titles.