How GPT-5 helped mathematician Ernest Ryu solve a 40-year-old open problem

Every significant math problem has a story—someone who posed a question, someone who tried to solve it, someone who could not, and eventually, maybe, someone who could. The story behind answering one frustratingly simple optimization theory question⁠(opens in a new window) is no different, except the researcher worked with a tool capable of quickly surfacing ideas and techniques from across a wide range of mathematical papers.

With 15 years in applied mathematics and optimization theory, Professor Ernest Ryu of the University of California, Los Angeles (UCLA), was curious about the large language model (LLM) everyone was talking about. In 2023, he decided to test ChatGPT‑3.5’s ability to solve simple math and logic problems, like scheduling meetings with multiple people across time zones. He noticed it would understand implicit constraints (like no one wanting to have a meeting between 12 a.m.–6 a.m.) but its ability to produce accurate results widely varied. It had many strengths, but in his opinion, it still had a long way to go.

When OpenAI unveiled GPT‑5 two years later, Ryu began hearing about its rapidly advancing capabilities in mathematics. He decided to try again now that the model had matured to see if it could handle a more complex problem. He did not expect that this would meaningfully contribute to solving a longstanding question in his field.

Ryu decided to tackle an “open” problem, meaning it was unsolved and recognized in the community as something of interest. His mathematical intuition told him it may admit a simple solution; a human just wasn’t able to find it yet.

The question: when an algorithm uses a phenomenon called the Nesterov Accelerated Gradient, or NAG, it becomes dramatically faster—but does the extra momentum added from NAG not affect the algorithm’s stability?

Unlike a car engine, which might fail if pushed too hard for too long, NAG seemed to deliver a “speed boost” without introducing instability. For decades, researchers observed this behavior, but they could not fully explain the underlying reason why the method remained both fast and stable.

NAG was first introduced in 1983 by mathematician Yurii Nesterov and is an optimization method that uses a form of prediction—often described as “looking ahead”—to speed up how quickly an algorithm converges. Unlike traditional algorithms, which take incremental steps and calculate a gradient—or the direction and steepness of a function's slope at a given point—NAG calculates the gradient at a look-ahead point, enabling algorithms to make a more informed final update. By anticipating where a parameter will be, as opposed to its current state at the time of calculation, NAG helps guide the algorithm’s steps more effectively, converges faster, and manages its inherent oscillations as it approaches the function’s minimum, resulting in quicker progress toward a solution.

To put it more simply, optimization theorists wondered why an increase in an algorithm’s momentum did not significantly affect its stability. In training machine learning models or solving engineering problems, efficiency is critical to avoid wasting computational resources and producing slower results.

“By expanding the theoretical tools available in optimization theory, we collectively design algorithms optimized for efficiency, stability and safety,” Ryu said. He and several others had tried solving this mystery years before but failed to produce a mathematical proof that uncovered the solution.

As Ryu continued prompting GPT‑5, often into the late hours of the night after his kids went to sleep, he was impressed with its creativity and unconventional approaches in attempting to solve this fundamental problem. However, as he double-checked the model’s work, he would notice mistakes in its reasoning. The problem remained unsolved.

“It had an interesting approach that I had not thought about,” he said—just like the human minds he valued so much in mathematical brainstorming. “So, that’s why I decided, okay, let me push this further.”

Throughout this process, GPT‑5 was not inventing new mathematical tools and principles. It was very good at wielding existing tools and finding equations, solutions and ideas from papers slightly outside the given area of expertise, which Ryu might not have come across otherwise. “It astonished me with the weird things it would try. Its ability to pull from this massive scale of reading and learning is what makes it really powerful.”

Ryu kept prompting, all the while treating GPT‑5 as a collaborator he would bounce ideas off of. He noticed it would continue to produce creative ideas that pushed him in new and unexpected directions. When he posed a question, it would offer a direction—right or wrong—and Ryu would assess it quickly. If the idea felt like a dead end, he would pivot immediately; if it showed promise, he would pursue it to see where it led him. The sheer speed of this process condensed what would normally take days into hours.

“GPT‑5 was a very unusual collaborator,” he said, “in that it would propose something completely out of the blue.”

GPT‑5 enabled Ryu to consider new potential paths for solving the proof that he would not have otherwise thought about, whether because he did not see a connection or because they were from an adjacent field of mathematics he was not as familiar with. As Ryu continued to explore ideas with GPT‑5, one thing became more and more apparent: these systems can be powerful exploratory tools when paired with subject-matter expertise and careful verification.

“The way math research works is you have ideas, and whenever you or your colleagues come up with a rough idea, you have a sense of whether or not it is going to work, like a mathematical intuition,” Ryu said. “This is where the partnership between artificial intelligence and humans can work especially well.”

He continued this approach of prompting the LLM for new ideas, ruling out the ones he felt were dead ends and chasing the ones he felt had potential.

Additionally, the speed at which he and GPT‑5 generated, evaluated, explored and exhausted ideas was markedly faster than it was on his own. After around twelve hours over three days of collaboration with GPT‑5, and nearly a dozen approaches later, one of the methods he was investigating finally worked.

This turning point emerged when the model proposed a way of restructuring the equations that govern the NAG method. The suggestion was not correct as written, but Ryu recognized the structural feature that seemed meaningful, developed it more rigorously on his own, and began asking the model targeted questions to test the potential proof’s viability. That line of inquiry ultimately became the backbone of the final proof he wrote out himself.

“It was a surreal feeling,” he said, equating the experience to working with a competent student who would come up with ideas, ask questions, and brainstorm. “It was really intellectually exciting to work with GPT‑5,” Ryu said. “I had to be vigilant in checking the calculation steps, but it was still much faster than doing the calculation myself.”

While the process was significantly faster, it also required careful verification. GPT‑5 often produced arguments that looked plausible but did not hold up on inspection. Ryu reviewed each suggestion, discarded the ones that failed, and developed the ones that contained a mathematically meaningful idea. The workflow depended on his judgment—both to identify promising directions and to recognize when a line of reasoning had been fully explored.

Ryu used GPT‑5 like a tool, and he took great care to understand what it was doing so he could discern what it was good at and what it wasn’t. When asking GPT‑5 to check the work, Ryu found greater success if he started a new chat as opposed to asking the model to check its work in the same chat. His process of feeding results into fresh conversations helped minimize accumulated errors. Then, he said, “you can sit down and carefully check all the details yourself,” like would normally be done in the research process.

Each session, Ryu would formulate a question, prompt GPT‑5 for a direction or technique and receive back a mathematical argument—sometimes correct, sometimes flawed, sometimes completely unexpected. While several of the arguments GPT‑5 generated were incorrect, it excelled at doing what humans would normally find cognitively exhausting: rapidly proposing and discarding variations of the same idea, searching the space of possible arguments, and borrowing mathematical tools from adjacent subfields to exhaustively search for answers. According to Ryu, “several of the key steps that ultimately mattered were suggested by GPT‑5,” even though it could not assemble them into a complete proof on its own.

The dynamic resembled exploring a massive maze with a companion who could reveal new paths instantly. Most paths led nowhere, but the speed of exploration meant dead ends were discovered more quickly. In classical research workflows, ruling out an idea can take substantial time.

The major benefit of utilizing GPT‑5 in this exploration was the acceleration the model provided in the overall process. The speed at which it can pull from a vast literature database allowed Ryu to rapidly rule out dead-end approaches and pursue promising ones. In about twelve hours, Ryu had solved what he estimates could have taken weeks of focused work, had he decided to pursue it at that pace.

“But that was never going to happen,” he said. “Because after three days of trying hard, I would have given up. As a mathematician, you have to make a call as to whether to give up problems and move on to the next thing.”

But with GPT‑5, he had the sense of making rapid progress, which kept him going. In his view, this created a psychological shift: the steady flow of new ideas made the problem feel within reach longer than it might have otherwise.

As a professor, Ryu takes the relationship between AI and learning very seriously. It is a tool that has to be understood and harnessed to get value out of it. Anyone using AI needs to be fluent enough with the subject they are exploring to know when the model is coming up with good ideas, when it is coming up with incorrect ideas and when it is making mistakes.

The pre-print paper on Ryu’s work to solve the NAG problem is available publicly⁠(opens in a new window) and is working its way through the peer review process, which takes around 12-18 months. In an effort to appeal to the broader mathematical community, Ryu wrote his paper in the traditional style. GPT‑5 came up with several key ideas that shaped the final proof, but for the paper, Ryu cleaned it up and wrote it himself, along with the associated narrative. GPT‑5 is mentioned in the title and abstract, and its contributions are explained throughout the paper. After thinking deeply about how to credit GPT‑5’s collaboration with helping to solve this proof, he ultimately decided it was used as a tool, which is why GPT‑5 is not listed as a co-author. 

“I want this to be a case study in informing classical mathematicians of the potential value of using GPT‑5. In order to do that, I felt I should probably conform to the classical style of publishing journal papers and this way of doing scholarship,” Ryu said.

Ryu’s experience is just one example of how GPT‑5 can help the world’s brightest minds find solutions to humanity’s most complex and intriguing questions. His success relied heavily on his expertise in the field of mathematics and his willingness to double-,triple-, and quadruple-check the LLM’s reasoning and assertion every step of the way.

Ultimately, Ryu thinks it will be a good thing when researchers use AI in productive and positive ways. He expects the diversity of ideas from adjacent fields to become more accessible, which he sees as a positive development.

“Be patient,” Ryu said. “If you want the model to fail, it will fail. But if your mindset is that you want to work with the model to extract value, then it’s much more likely that you will.”

“This experience really had a profound influence on me. I plan to use AI in my math research all the time going forward,” Ryu said. “There's no reason why I wouldn’t.”