OpenAI GPT-3: How Generative Code Actually Started
OpenAI GPT-3 proved that raw scale produced general language ability but not working software. Functional code generation required a separate, code-specific fine-tune, and that model, Codex, went on to define the AI pair programmer product category through GitHub Copilot.
Table of Contents:
- Key Takeaways
- Introduction
- What Was the Big Bet Behind GPT-3?
- Why Did OpenAI's Best Model Score 0% on Code?
- How Did 159 GB of GitHub Python Create Codex?
- How Did GitHub Copilot Turn Codex Into a Daily Habit?
- Where Did OpenAI Codex Break Down?
- What Did the Codex Era Leave Behind for Self-Taught Makers?
- Frequently Asked Questions
Key Takeaways
- A model that beat the average college applicant on SAT analogies could not solve a single beginner Python problem, and that contradiction explains everything that followed.
- The jump from unusable to useful took no new architecture and no breakthrough. It took 159 gigabytes of filtered GitHub Python and a tokenizer change nobody outside the lab noticed.
- The class action filed in November 2022 was not really about stolen code. It was about attribution, and the question it raised is still open.
Introduction
In May 2020, OpenAI released a language model with 175 billion parameters that outscored the average human college applicant on SAT analogy questions and could not write one working Python function. If you are a self-taught maker who ships side projects with an AI assistant sitting in your editor, the two-year gap between those two facts is your origin story.
The interesting part is not that the model got smarter. It is that the fix was narrow, cheap by comparison, and almost boring: a filtered pile of public code plus a tokenizer tweak. Everything after, Copilot, vibe coding, the lawsuits, the agents, follows from that one correction.
Here is the full sequence, dated and sourced, from a 0.0% benchmark score to the tools you probably used this week.
What Was the Big Bet Behind GPT-3?
GPT-3 was a bet that scale could replace task-specific training. According to Brown et al. in the May 2020 GPT-3 paper, the 175-billion-parameter model represented a tenfold jump over any prior non-sparse neural network, and it learned new tasks from a handful of examples typed into the prompt, with no gradient updates and no labeled dataset.
Before this, the recipe was pre-train on text, then fine-tune on thousands of labeled examples for every task you cared about. According to Brown et al., that approach scored well and broke in predictable ways: it demanded a fresh dataset per application, it latched onto spurious correlations inside narrow training distributions, and it generalized poorly outside them.
Tom B. Brown and 30 co-authors tested the alternative on roughly 500 billion tokens. The sampling was deliberately lopsided (Brown et al., May 2020):
- Filtered Common Crawl: 410 billion tokens, 60% of the blend, sampled at 0.44
- WebText2: 19 billion tokens, 22% of the blend, sampled at 2.90
- Books2: 55 billion tokens, 8% of the blend, sampled at 0.43
- English Wikipedia: 3 billion tokens, 3% of the blend, sampled at 3.40
Wikipedia is a rounding error in that pile by raw volume, and the training run went through it 3.4 times. The paper's real claim was not that the model knew more than its predecessors, but that it needed to be told far less.
The benchmark results backed the thesis. According to Brown et al., GPT-3 hit 71.2% few-shot accuracy on closed-book TriviaQA, 82.8% few-shot on the PhysicalQA common-sense reasoning set, and 65.2% on SAT analogies against an average human applicant score of 57%.
Why Did OpenAI's Best Model Score 0% on Code?
Because prose and programs are graded differently. According to Chen et al. in July 2021, base GPT-3 solved 0.0% of problems on the 164-question HumanEval benchmark, at every parameter size and sampling temperature tested. Text can be approximately right and still read fine. Code either runs and passes its unit tests or it does not.
That grading choice was the real contribution. Earlier code research leaned on surface-similarity metrics like BLEU, which reward output that looks like the reference answer. OpenAI threw that out and introduced pass@k: generate k samples, execute them, and count the problem solved only if at least one passes every test.
A model can produce code that reads beautifully, imports the right libraries, names its variables sensibly, and still fails every test. BLEU cannot see the difference. Unit tests can.
I think this is the most underrated moment in the whole story. Everyone remembers the parameter count, but the benchmark redefinition is what made progress measurable, and what makes today's vibe coding workflows evaluable at all.
Smaller open models were already scoring above zero on the same test, which kills the easy explanation that code was simply too hard in 2020:
- GPT-Neo, 2.7B parameters: 6.4% pass@1 (Chen et al., July 2021)
- GPT-J, 6.0B parameters: 11.4% pass@1 (Chen et al., July 2021)
- Base GPT-3, 12B to 175B parameters: 0.0% pass@1 (Chen et al., July 2021)
A 6-billion-parameter open model beat a 175-billion-parameter flagship by 11.4 points. Scale was not the variable that mattered here. Diet was.
How Did 159 GB of GitHub Python Create Codex?
The fix started in May 2020, months before anyone outside the lab knew there was a problem. According to Chen et al., researchers scraped 179 gigabytes of unique Python files from 54 million public GitHub repositories, then filtered out auto-generated code, files averaging over 100 characters per line, and files with minimal alphanumeric content, leaving a clean 159-gigabyte corpus.
There was also a quieter change. Standard byte-pair encoding splits whitespace inefficiently, so Python indentation was burning context window for nothing. According to Chen et al., adding explicit tokens for whitespace runs cut the tokens needed to represent Python source by roughly 30%.
"Evaluating Large Language Models Trained on Code" by Mark Chen and colleagues landed on July 14, 2021. The results, all from that paper:
| Model | Parameters | pass@1 | pass@100 | Reranked pass@1 |
|---|---|---|---|---|
| Base GPT-3 | 12B to 175B | 0.0% | 0.0% | Not applicable |
| GPT-Neo | 2.7B | 6.4% | 21.3% | Not applicable |
| GPT-J | 6.0B | 11.4% | 27.7% | Not applicable |
| OpenAI Codex | 12B | 28.8% | 70.2% | Not applicable |
| Codex-S (fine-tuned) | 12B | 37.7% | 77.5% | 44.5% |
Read the last row twice. One attempt from Codex-S solved 37.7% of problems, but 100 attempts at a higher temperature, filtered by unit tests, solved 77.5%. According to Chen et al., when unit tests were unavailable, picking the sample with the highest mean token log-probability still reached 44.5%.
How Did GitHub Copilot Turn Codex Into a Daily Habit?
A benchmark score is not a product. According to GitHub's launch announcement, CEO Nat Friedman announced the technical preview of GitHub Copilot on June 29, 2021, two weeks before the Codex paper published, shipping the model as a Visual Studio Code extension rather than an API endpoint developers had to wire up themselves.
GitHub Copilot draws context from the code you're working on, suggesting whole lines or entire functions.
That distribution choice mattered more than the model. According to GitHub, the deployment shifted the developer role away from manual syntax entry toward code review, system design, and prompt editing, with engineers generating snippets inside the IDE instead of switching tabs to hunt for API usage examples.
The competitive response arrived fast. According to DeepMind, AlphaCode shipped in February 2022 aimed at Codeforces competitive programming, using mass sampling and execution-based clustering rather than real-time autocomplete. Two philosophies, one shared insight: generate many candidates, then filter.
Classrooms felt it too. According to research published on arXiv in April 2026, generative tools improved novice execution speed and reduced programming anxiety, while forcing educators to rebuild assessments around code verification, edge-case testing, and debugging instead of basic syntax. That is roughly the same skill shift facing anyone learning to build through no-code AI app builders today.
Where Did OpenAI Codex Break Down?
Codex inherited its training data's bad habits along with the good ones. According to Chen et al. in July 2021, safety evaluations found insecure code patterns in roughly 40% of tested scenarios, faithfully reproducing vulnerabilities that were already sitting in public repositories.
The specific failures documented in the Codex paper and contemporaneous coverage:
- Broken cryptographic primitives, including RSA used without OAEP padding (CWE-780)
- Security misconfigurations mapping directly to OWASP Top 10 categories
- Single-precision floats representing financial values, a flaw that appeared in early Copilot promotional material according to AI Business in July 2021
- Variable binding failures, with the model hallucinating uninitialized variables or calling out-of-scope functions across longer contexts
Then there is the reasoning ceiling. According to OpenAI's research, when docstring prompts were modified to chain multiple functional requirements together, the pass rate degraded by a factor of two to three for each additional operation added. Two chained steps roughly quartered performance. Four became close to hopeless.
Codex was excellent at the function and helpless at the system, which is precisely the boundary separating autocomplete from engineering.
The 2,048-token context window locked analysis to isolated single-file functions, so the model could not see your other files even if it wanted to. If you ship anything handling payments or user data, this history is why a security review of no-code and AI-assisted builds still belongs on your checklist.
What Did the Codex Era Leave Behind for Self-Taught Makers?
Two inheritances: an unresolved legal question and a completely different tooling stack. In November 2022, attorneys Joseph Saveri and Matthew Butterick filed the class action Doe v. GitHub, Inc. et al. in the U.S. District Court for the Northern District of California against Microsoft, GitHub, and OpenAI.
The claim was narrower than most coverage suggested. According to court filings, the allegation centred on Section 1202(b) of the DMCA: training on GPL, Apache, and MIT licensed repositories, then emitting code stripped of copyright notices, attribution, and license terms. Not theft of code, removal of credit.
when we filed that first case against Microsoft, we actually got death threats. They were serious enough that the judge allowed our plaintiffs to proceed anonymously.
That is Joseph Saveri, speaking to Super Lawyers about the filing. The scale of literal copying was small: according to GitHub's own documentation, roughly 0.1% of Copilot suggestions contained verbatim matches over 150 characters. Small percentages of very large numbers are still large numbers, which is why corporate legal departments kept flagging copyleft contamination.
The technical inheritance moved faster than the legal one. The filing landed in the same month as the ChatGPT release that broke Stack Overflow's traffic, and within two years the standalone Codex API was deprecated, its capabilities folded into instruction-tuned models. Copilot grew into an enterprise platform covering test generation, pull request synthesis, and multi-file workspace indexing across VS Code, JetBrains, and Neovim.
Everything downstream inherited the same design pattern. BigCode's StarCoder and Meta's Code Llama gave privacy-sensitive teams hostable weights, Google's own code generation play arrived through AI Studio, and the browser-based builders like Lovable, Bolt and v0 turned prompt-to-app into a consumer product.
Does the provenance question change how you should build? Depends entirely on what you are shipping. For a landing page or a client portal, it is close to academic. For anything you plan to license, sell, or take through due diligence, knowing what your generator was trained on is not paranoia, it is hygiene, and the difference between no-code and vibe-coded builds matters most exactly here.
Start exploring launch-ready no-code AI agent templates here!
Frequently Asked Questions
What is OpenAI Codex?
Codex is a model family fine-tuned from GPT-3 on public Python repositories, introduced by Mark Chen and colleagues on July 14, 2021. It translated natural language docstrings into executable code and powered the first version of GitHub Copilot before being retired as a standalone API.
What does the HumanEval benchmark measure?
HumanEval is a set of 164 hand-written Python problems released by OpenAI to test whether generated code actually runs. Unlike text-similarity scores, it executes each candidate against unit tests, so a solution counts only when it passes every test for that problem.
Is GitHub Copilot the same thing as Codex?
No. Codex was the underlying model, and Copilot is the product Microsoft and GitHub built on top of it, launched in technical preview on June 29, 2021. Copilot has since moved to newer models and expanded well beyond the inline autocomplete it started as.
What does pass@k actually mean?
Pass@k measures the probability that at least one of k generated attempts passes all unit tests. It matters because repeated sampling works: Codex-S solved 37.7% of HumanEval problems in a single attempt and 77.5% across 100 attempts, according to Chen et al.
Is AI-generated code safe to ship without review?
Not without checking it. OpenAI's 2021 evaluations found insecure patterns in roughly 40% of tested scenarios, including broken cryptography and OWASP Top 10 misconfigurations. Static analysis and human review on anything touching authentication or payments remains sensible, as 2023's thin AI wrappers demonstrated.
Start building without code
Browse thousands of no-code templates for Webflow, Framer, Bubble, Lovable, Replit and more.
Explore Templates
