Thank you for putting into words many anxieties I have with regards to the future of mathematics in the AI era. This is also brought to my attentions multiple facets I've not considered, in a very thoughtful way.
I've my own misgivings about using AI. I've tried (unsuccessfully so far) to use it in research and was thinking about delving deeper, motivated by the recent discovery of a non-sofic group, based on the work of Kun and Thom. Your post definitely cooled this fire, to my own relief.
My completely novel and original take is that there are good and bad ways to use this technology. I've not experimented much and thus have not drawn my own line of where the distinction lies. I think bad ways are the one which reduce our understanding, autonomy and agency in mathematics. This is your run-of-the-mill ''do my homework'' prompt.
I believe there are good ways to use it in teaching yourself and other known mathematics. An eye-opening example from a few years ago was (IIRC) Eric Hogle's talk'Peer-editing vs chatGPT' (I will put links in PS) at JMM2024. Going by memory, the speaker mentioned teaching an intro to proof course, and in one exercice, instead of asking students to write a proof and to judge the proof of a partner, they were asked to generate chatGPT proof and to correct them. The students gave sharper critique of the AI proof than they would to a peer, which I found very interesting. I would categorize this as a good use of AI for mathematics.
(Ironically, I had to use chatgpt to refind the sources!)
I think it also makes sense to draw parallels with software engineering. This is a also a technical, but objective (in the sense that the code *must* compile) field, which fell victim to AI before mathematics. I think there is a cautionary tale there. The most scary thing is the ''lost generation'' of software engineer, a few years where there was virtually no hire, under the assumption they could be replaced with AI. This removed the bottom rung of the career ladder for many young people.
In retrospect, coding agents have taken much less place than expected, with the cost of attention for long projects overshadowing the price of an entry-level programmer. But the most adverse effect is that by not hiring a new generation, the ''talent'' pipeline has effectively been cut.
This may be a naive understanding of the situation from an outsider (and I would love to be corrected). I fear something similar may happen in mathematics, as you mention in your ''pipeline'' point.
As comedic relief, I'd like to drop some of my ''favorite'' AI shenanigans.
1. By manipulating headline and changing certain letters, for example to cyrillic letters, stock-trading AI can be mislead to misevalute certain companies: https://arxiv.org/html/2601.13082v1
(oh no, the poor algorithmic stock traders! :'( )
2. By fine-tuning an AI model to write insecure code, AI can just become evil (I believe the technical term is 'misaligned') https://arxiv.org/abs/2502.17424
lots of interesting ideas to take away from this; i'd like to focus on the question of why do we do mathematics in the first place. i agree with the answer you give, and i find comfort in Michael Harris' characterization of mathematics as 'a way of being human', instead of a theorem-proving industry. unfortunately, in practice, it is often degraded to the latter. even before this recent influx of AI-generated mathematical results, a lot of us have already been feeling disillusioned with the 'publish or perish' culture in academia. the emphasis on producing results (as opposed to playing freely with mathematical ideas) because your career depends on it can be stifling to mathematical creativity, and is one of many many reasons PhDs are leaving academia en masse. my hope is that these unprecedented times force us into some long overdue reflection on systemic questions about how mathematical research is done.
I will discuss only research implications (teaching I think may be a separate problem).
Let me start discussing yesterday's proof: it is not new. I'm trying to be a bit provocative, but as far as my understanding of the proof and area goes, the proof gathers results experts were already aware of and merges them in an argument that is not "unexpected". Unexpected is what mathematicians bring to the table.
I think llms are a great power that we can use as mathematicians. Because, as a former analyst/probabilist, I did struggle to and avoid to read, for example, Jaikin-Zapirain's papers (for them being far from my area of understanding, all my respect to an extraordinary mathematician), but the AI tools can help one cross this bridge between areas.
In my opinion, mathematicians have a great tool at hand that will not remove the heavy-lifting but just avoid the tasks humans are too lazy to do.
But let's go to the materialism:
1. We need not depend on corporations. There are very solid open source models that universities can host locally. Universities can also use resources in tuning these models that remain open source.
The technology is extraordinary. It just happens that it's a succulent bubble for the Trumpist world we live in (the west) and there is no interest in developing this technology from the public sector.
2. These AI proofs are exposing a pre-existing problem. Math has been (let me use the word xd) poisoned by turbocapitalist/publish-or-peril where what matters is the number of papers full-stop. Because in the end mathematicians eat from tenure, not recognition.
There are already a lot of questionable papers around that noe with LLMs are just trivial. What should be adressed is how mathematics (and academia as a whole) measures success. Now we are measuring success in a way that llms are harmful. But the system itself was in a way not aligned with math. This is an opportunity to rediscuss the system.
To close up, the way we do mathematics is very recent. In our area if you want to go veeery far it's not even a 100 years (some von Neumann papers?). Math has undergone constant change, and this may just be the trigger for the next change. We should embrace the technology and discuss the "systemic" questions of how to integrate it.
Maybe this is just me, but I feel like there's mismatch between the described harm and the ask.
Reading the three ways that AI is at odds with the values of the mathematical community:
1. LLMs don't do a great of at credit attribution. Okay, I believe this but of the possible ways to address it, it seems like improving the model's attribution skills is the better path to take? In other words, this reads more like a feature request than a damning criticism. No matter what, humans can do the work to fill in the attributions that the model dropped the ball on.
And let's not forget, LLMs also ferret out work that has been long forgotten by humans and surfaces it. It's not clear how to weigh those two things against each other. But either way, solving it sounds like an engineering problem. And re: those attribution details that are generally de-emphasized in press write-ups? Well, that's much more about press writeups than AI. It happens with human results as well, doesn't it?
2. LLMs frequently write nonsense proofs. Okay, what's the argument here though? This feels either self-resolving or unimportant.
If they generate so much nonsense that using them is net-negative, then problem solved. No one will use them.
If they're net-positive, then why does this matter? If anything, it suggests a vector along which the models will likely improve rapidly. If the sense-to-nonsense ratio is high enough to warrant their use, then why is this an argument against using them?
3. They incentivize secrecy. Perhaps, but I think what they will really do is change the kind of problem that feels interesting. I imagine a similar thing happened when computer assisted proofs starting going mainstream. Suddenly, a certain class of problem was low-hanging fruit. No longer of interest.
The problems that you imagine people will become secretive about - namely, the ones that an LLM makes it easy to solve - will become uninteresting very quickly for precisely the reason that LLMs are knocking them off left and right. The fruit higher in the tree will become the new floor for interesting. Maybe?
I could easily be missing something her and if so, I'm all ears.
But if not, the harms described don't seem to rise to the level of a call to arms against AI. Nor does it seem like it would work. I mean if the possibility of a global-scoped AI apocalypse isn't enough to slow us down then I'm not sure these criticisms can move the needle.
I think there are lots of mathematicians at mid-level institutions (not trying to be elitist, but do need to get this point across) with unfinished drafts, projects, prospective experiments that they have accumulated throughout their careers. These projects won't be handed off to grad students or colleagues and in many cases are not suitable/fleshed out/high payoff enough for that or are really only of primary interest to one or two mathematicians. Telling them not to use AI to make progress on their niche little research areas that they've tossed around for decades seems wrong (and cruel?). People's lives are finite after all.
Let me also point out that when a significant result gets proved then at least in the short term understanding, digesting, and following up on its consequences is a worthwhile endeavor that invigorates the field, and at least for now that can be done by humans rather than ai: sofic groups are not my area but they are not so far off, and I do find the directions this result opens up quite exciting. It is now worthwhile to look at conjectures that are proved for sofic groups and try to find counterexamples to them in general.
-In dynamics, there is Gottschalk's surjunctivity conjecture (initial reason Gromov introduced the class of groups as ``those that i an prove the conjecture for''),
-In low dimensional topology/group theory/equations-over-groups there is the Kervaire-Laudenbach conjecture, saying that an acyclic 2-complex with non-trivial fundamental group G does not embed in a contractible 2-complex. It is known when G is sofic or more generally hyperlinear, but not beyond that.
-In group theory/L^2 invariant land, there is an approximation theorem computing L^2-Betti numbers from finite data for sofic groups, so understanding L^2-Betti related questions for these non-sofic examples might be interesting. Closely related, there is also the ``determinant conjecture'' about Fuglede-Kadison determinants that is known for sofic groups but not in general.
-One can also try understand the proof and see if it gives hints about whether some other candidates that are closer related to lattices are nonsofic.
In general, it shows that the world is more interesting than we thought.
It establishes another instance of Gromov's metaconjecture ``Every statement about all discrete groups is either trivial or false.''
P.S.: I don't know much about this, but when quantum information people disproved the Connes embedding conjecture a few years ago, there was some hope that it would lead to a non-sofic group. Maybe there will be some feedback in the other direction, i.e. do these groups say anything about quantum information? (This is beyond my paygrade...)
While I commiserate, I’m not handicapping myself in undergrad. There is a distinction between use of chat-tier LLMs for knowledge work and fracking for new theorems using frontier AI.
I think your prognosis on the future of mathematics is unfortunately roughly right. I always wanted to be an applied research mathematician, and currently the most promising path that I can see is in ML/AI. I’m still sleeping with one eye open.
"I believe that today, mathematicians may be in the same position as these bygone professions."
> Do you believe that these 10 things are serious? It appears that if you think that the clankers are about to get 'there'. How serious is this really? How real is this result? "many big conjectures to fall in the next year" Do you think that these are the real serious things? How certain are you? These directly imply eventual autonomy to me. Why should a robot be able to discover a proof yet fail to verify it?
You call ai "I believe that access to an increasingly-powerful oracle..."
> I think you're saying that if you have 100% of your time to spend, if the clanker is spitting out things for you to verify by hand, you might as well spend your time checking its work.
The existence of a technology is not an obligation to use it.
> I agree because in the year 2021 before clankers were born, people would sort things out by mail and will continue to do so. Sometimes i might consider the thought of or enjoy writing a letter myself. Not only that, not everyone has learned to use a computer today.
"There is no denying that AI companies have made spectacular scientific advancements: this means we are in the domain of politics now."
> This was great because it gave me quite a bit to reflect on.
Let me restate my understanding of what you've said. AI is disruptive to mathematical understanding and in short, money is flowing into it. I've been told that clanker-building companies are paying people quant shop salaries to do math for them.
> This will be one eventual tactical though not strategic challenge for you if you intend to 'negotiate'.
And I don't think it's about negotiation. Negotiation implies power but counted. If these companies want to, they'll train their own people from scratch. It probably won't go there. But our demands in these negotiations are ideals that we may not achieve in the end. Negotiation in any idealized form either can be achieved or not. A different approach is to consider the norm you're searching to establish. You need to bring others, like the companies, either into alignment or discussion on this norm. Focus on establishment of it instead of negotiation as the end result. The mathematics person is brought to understand and align with the norm. The regular person who works a 9-5 job at a grocery store may decide in favor or against of this norm. The ai people will obviously oppose construction of this norm but at least the focus is to shift it towards a direct dispute about what these norms should be. For instance the idea is Dario will have to say "I believe that these axioms should guide mathematics instead." This is what you want. It's better to search for an overshoot instead of to try to scramble together to negotiate.
You mention undergraduate performance dropping
> I'd like to just mention my own strong feelings/observation, clankers are only helping people with homework/doing it for them because it makes the clanker builders money and a consistent user base.
"it increasingly seems that artificial intelligence technologies are autonomously capable of doing the same"
> The obvious weakness of OAI's results are that
1. It's mightily suspicious that this run is claimed to have cost 1,000 API USD. That's getting close to free.
2. If 10 results, why not 100? 1,000,000? On team anthropic just having a clanker doing the cryptographic weakness was a big deal. Now two models that I suspected were neck-and-neck have totally different results.
3. If this product is in internal previews right now, finding 10 results seems quite fast. if more doesn't come quickly, i have reason to become suspicious of inconsistencies or issues thatr range from cherrypicking to something worse.
Do you have any thoughts here? If 10 open problems, then why not research how to build better clankers instead? Why not do more math on ai instead of this? This appears to imply the underlying strategics to me, which is that attention and flair is more important at this moment than driving more frontier LLM research?
"I believe that in a world where AI-led mathematics directly competes with human-led mathematics, AI-led mathematics will ultimately surpass us."
> Maybe. I read your link to Terence Tao's compiled ideas and he suggests that math can move and change over time. Supposedly we can't predict what work will be like. That being said, this is concerning.
"Perhaps the future career of the mathematician is that of a prompt engineer"
> Of course everything we're thinking of and panicking about in these timelines collapses any length of time into a brief moment. If it will kill mathematics, our concerns are now. If it will automate your job away or kill you one day, your concerns are now. If the mathematician is just a prompt engi, i'm implying this can't be true since the very first job that you would one day want to automate away if you're a company is prompt engineering, since you're just putting in a question. Clearly, the fact that there is leverage on the prompting side by the humans reflects either the reality or lack of innovation for companies to move there yet. This feels quite firm to me.
In the end this is a conceptually interesting proposition and it exists at the maximal edge between model capability and human capabilities. This is about clankers vs. humans, except with a sardonic twist. The humans are driving clanker development. When I say maximal, I think to myself that the only exception to this argument is if human cognition goes on to explore areas that LLMs just can't compete in one day and some weird job or means of sustainment and creation is invented.
I think your precision comes where you state that the world being created is just not one you want to live in.
Human ingenuity isn't dead yet. There are scaling laws and AI doesn't scale linearly. That is why OpenAI needs 1/2 a trillion dollars and a nuclear reactor to reach your level of intelligence. The real question is "when will AI develop good taste." That means "when will it pick good problems to solve" and not "when will it solve problems." Because right now it is hitting targets being held up by humans by emulating human created datasets. Once AI starts identifying research questions better than a human on its own that improve AI, working from datasets it created then we are all peasants. Right now it isn't clear that we can keep scaling AI much further and there may be a reprieve for our species.
Thank you for putting into words many anxieties I have with regards to the future of mathematics in the AI era. This is also brought to my attentions multiple facets I've not considered, in a very thoughtful way.
I've my own misgivings about using AI. I've tried (unsuccessfully so far) to use it in research and was thinking about delving deeper, motivated by the recent discovery of a non-sofic group, based on the work of Kun and Thom. Your post definitely cooled this fire, to my own relief.
My completely novel and original take is that there are good and bad ways to use this technology. I've not experimented much and thus have not drawn my own line of where the distinction lies. I think bad ways are the one which reduce our understanding, autonomy and agency in mathematics. This is your run-of-the-mill ''do my homework'' prompt.
I believe there are good ways to use it in teaching yourself and other known mathematics. An eye-opening example from a few years ago was (IIRC) Eric Hogle's talk'Peer-editing vs chatGPT' (I will put links in PS) at JMM2024. Going by memory, the speaker mentioned teaching an intro to proof course, and in one exercice, instead of asking students to write a proof and to judge the proof of a partner, they were asked to generate chatGPT proof and to correct them. The students gave sharper critique of the AI proof than they would to a peer, which I found very interesting. I would categorize this as a good use of AI for mathematics.
https://meetings.ams.org/math/jmm2024/meetingapp.cgi/Paper/31913
https://www.gonzaga.edu/news-events/stories/ai-around-campus?utm_source=chatgpt.com
(Ironically, I had to use chatgpt to refind the sources!)
I think it also makes sense to draw parallels with software engineering. This is a also a technical, but objective (in the sense that the code *must* compile) field, which fell victim to AI before mathematics. I think there is a cautionary tale there. The most scary thing is the ''lost generation'' of software engineer, a few years where there was virtually no hire, under the assumption they could be replaced with AI. This removed the bottom rung of the career ladder for many young people.
In retrospect, coding agents have taken much less place than expected, with the cost of attention for long projects overshadowing the price of an entry-level programmer. But the most adverse effect is that by not hiring a new generation, the ''talent'' pipeline has effectively been cut.
This may be a naive understanding of the situation from an outsider (and I would love to be corrected). I fear something similar may happen in mathematics, as you mention in your ''pipeline'' point.
As comedic relief, I'd like to drop some of my ''favorite'' AI shenanigans.
1. By manipulating headline and changing certain letters, for example to cyrillic letters, stock-trading AI can be mislead to misevalute certain companies: https://arxiv.org/html/2601.13082v1
(oh no, the poor algorithmic stock traders! :'( )
2. By fine-tuning an AI model to write insecure code, AI can just become evil (I believe the technical term is 'misaligned') https://arxiv.org/abs/2502.17424
3. AI fails very basic understanding on simple rephrasing of famous riddles : https://www.reddit.com/r/singularity/comments/1fqjaxy/contextual_training_and_overreliance_on_llms/
lots of interesting ideas to take away from this; i'd like to focus on the question of why do we do mathematics in the first place. i agree with the answer you give, and i find comfort in Michael Harris' characterization of mathematics as 'a way of being human', instead of a theorem-proving industry. unfortunately, in practice, it is often degraded to the latter. even before this recent influx of AI-generated mathematical results, a lot of us have already been feeling disillusioned with the 'publish or perish' culture in academia. the emphasis on producing results (as opposed to playing freely with mathematical ideas) because your career depends on it can be stifling to mathematical creativity, and is one of many many reasons PhDs are leaving academia en masse. my hope is that these unprecedented times force us into some long overdue reflection on systemic questions about how mathematical research is done.
Hi Tasmin, very nice article!
I will discuss only research implications (teaching I think may be a separate problem).
Let me start discussing yesterday's proof: it is not new. I'm trying to be a bit provocative, but as far as my understanding of the proof and area goes, the proof gathers results experts were already aware of and merges them in an argument that is not "unexpected". Unexpected is what mathematicians bring to the table.
I think llms are a great power that we can use as mathematicians. Because, as a former analyst/probabilist, I did struggle to and avoid to read, for example, Jaikin-Zapirain's papers (for them being far from my area of understanding, all my respect to an extraordinary mathematician), but the AI tools can help one cross this bridge between areas.
In my opinion, mathematicians have a great tool at hand that will not remove the heavy-lifting but just avoid the tasks humans are too lazy to do.
But let's go to the materialism:
1. We need not depend on corporations. There are very solid open source models that universities can host locally. Universities can also use resources in tuning these models that remain open source.
The technology is extraordinary. It just happens that it's a succulent bubble for the Trumpist world we live in (the west) and there is no interest in developing this technology from the public sector.
2. These AI proofs are exposing a pre-existing problem. Math has been (let me use the word xd) poisoned by turbocapitalist/publish-or-peril where what matters is the number of papers full-stop. Because in the end mathematicians eat from tenure, not recognition.
There are already a lot of questionable papers around that noe with LLMs are just trivial. What should be adressed is how mathematics (and academia as a whole) measures success. Now we are measuring success in a way that llms are harmful. But the system itself was in a way not aligned with math. This is an opportunity to rediscuss the system.
To close up, the way we do mathematics is very recent. In our area if you want to go veeery far it's not even a 100 years (some von Neumann papers?). Math has undergone constant change, and this may just be the trigger for the next change. We should embrace the technology and discuss the "systemic" questions of how to integrate it.
Maybe this is just me, but I feel like there's mismatch between the described harm and the ask.
Reading the three ways that AI is at odds with the values of the mathematical community:
1. LLMs don't do a great of at credit attribution. Okay, I believe this but of the possible ways to address it, it seems like improving the model's attribution skills is the better path to take? In other words, this reads more like a feature request than a damning criticism. No matter what, humans can do the work to fill in the attributions that the model dropped the ball on.
And let's not forget, LLMs also ferret out work that has been long forgotten by humans and surfaces it. It's not clear how to weigh those two things against each other. But either way, solving it sounds like an engineering problem. And re: those attribution details that are generally de-emphasized in press write-ups? Well, that's much more about press writeups than AI. It happens with human results as well, doesn't it?
2. LLMs frequently write nonsense proofs. Okay, what's the argument here though? This feels either self-resolving or unimportant.
If they generate so much nonsense that using them is net-negative, then problem solved. No one will use them.
If they're net-positive, then why does this matter? If anything, it suggests a vector along which the models will likely improve rapidly. If the sense-to-nonsense ratio is high enough to warrant their use, then why is this an argument against using them?
3. They incentivize secrecy. Perhaps, but I think what they will really do is change the kind of problem that feels interesting. I imagine a similar thing happened when computer assisted proofs starting going mainstream. Suddenly, a certain class of problem was low-hanging fruit. No longer of interest.
The problems that you imagine people will become secretive about - namely, the ones that an LLM makes it easy to solve - will become uninteresting very quickly for precisely the reason that LLMs are knocking them off left and right. The fruit higher in the tree will become the new floor for interesting. Maybe?
I could easily be missing something her and if so, I'm all ears.
But if not, the harms described don't seem to rise to the level of a call to arms against AI. Nor does it seem like it would work. I mean if the possibility of a global-scoped AI apocalypse isn't enough to slow us down then I'm not sure these criticisms can move the needle.
I think there are lots of mathematicians at mid-level institutions (not trying to be elitist, but do need to get this point across) with unfinished drafts, projects, prospective experiments that they have accumulated throughout their careers. These projects won't be handed off to grad students or colleagues and in many cases are not suitable/fleshed out/high payoff enough for that or are really only of primary interest to one or two mathematicians. Telling them not to use AI to make progress on their niche little research areas that they've tossed around for decades seems wrong (and cruel?). People's lives are finite after all.
Cheers from a (former) fellow techer!
To be honest, I don't think it's wrong and cruel to tell people not to use AI to make progress on their niche little research areas.
Let me also point out that when a significant result gets proved then at least in the short term understanding, digesting, and following up on its consequences is a worthwhile endeavor that invigorates the field, and at least for now that can be done by humans rather than ai: sofic groups are not my area but they are not so far off, and I do find the directions this result opens up quite exciting. It is now worthwhile to look at conjectures that are proved for sofic groups and try to find counterexamples to them in general.
-In dynamics, there is Gottschalk's surjunctivity conjecture (initial reason Gromov introduced the class of groups as ``those that i an prove the conjecture for''),
-In low dimensional topology/group theory/equations-over-groups there is the Kervaire-Laudenbach conjecture, saying that an acyclic 2-complex with non-trivial fundamental group G does not embed in a contractible 2-complex. It is known when G is sofic or more generally hyperlinear, but not beyond that.
-In group theory/L^2 invariant land, there is an approximation theorem computing L^2-Betti numbers from finite data for sofic groups, so understanding L^2-Betti related questions for these non-sofic examples might be interesting. Closely related, there is also the ``determinant conjecture'' about Fuglede-Kadison determinants that is known for sofic groups but not in general.
-One can also try understand the proof and see if it gives hints about whether some other candidates that are closer related to lattices are nonsofic.
In general, it shows that the world is more interesting than we thought.
It establishes another instance of Gromov's metaconjecture ``Every statement about all discrete groups is either trivial or false.''
P.S.: I don't know much about this, but when quantum information people disproved the Connes embedding conjecture a few years ago, there was some hope that it would lead to a non-sofic group. Maybe there will be some feedback in the other direction, i.e. do these groups say anything about quantum information? (This is beyond my paygrade...)
P.P.S: There is now an explanation of the main proposition by Andreas Thom on mathoverflow (along with a link to a slightly different group by Francesco Fournier-Facio that is also proved to be non-sofic using this proposition). https://mathoverflow.net/questions/513866/what-are-the-key-new-ideas-in-the-proof-of-nonsoficity-of-groups-in-openai-s-con#comment1341500_513866
While I commiserate, I’m not handicapping myself in undergrad. There is a distinction between use of chat-tier LLMs for knowledge work and fracking for new theorems using frontier AI.
I think your prognosis on the future of mathematics is unfortunately roughly right. I always wanted to be an applied research mathematician, and currently the most promising path that I can see is in ML/AI. I’m still sleeping with one eye open.
"I believe that today, mathematicians may be in the same position as these bygone professions."
> Do you believe that these 10 things are serious? It appears that if you think that the clankers are about to get 'there'. How serious is this really? How real is this result? "many big conjectures to fall in the next year" Do you think that these are the real serious things? How certain are you? These directly imply eventual autonomy to me. Why should a robot be able to discover a proof yet fail to verify it?
You call ai "I believe that access to an increasingly-powerful oracle..."
> I think you're saying that if you have 100% of your time to spend, if the clanker is spitting out things for you to verify by hand, you might as well spend your time checking its work.
The existence of a technology is not an obligation to use it.
> I agree because in the year 2021 before clankers were born, people would sort things out by mail and will continue to do so. Sometimes i might consider the thought of or enjoy writing a letter myself. Not only that, not everyone has learned to use a computer today.
"There is no denying that AI companies have made spectacular scientific advancements: this means we are in the domain of politics now."
> This was great because it gave me quite a bit to reflect on.
Let me restate my understanding of what you've said. AI is disruptive to mathematical understanding and in short, money is flowing into it. I've been told that clanker-building companies are paying people quant shop salaries to do math for them.
> This will be one eventual tactical though not strategic challenge for you if you intend to 'negotiate'.
And I don't think it's about negotiation. Negotiation implies power but counted. If these companies want to, they'll train their own people from scratch. It probably won't go there. But our demands in these negotiations are ideals that we may not achieve in the end. Negotiation in any idealized form either can be achieved or not. A different approach is to consider the norm you're searching to establish. You need to bring others, like the companies, either into alignment or discussion on this norm. Focus on establishment of it instead of negotiation as the end result. The mathematics person is brought to understand and align with the norm. The regular person who works a 9-5 job at a grocery store may decide in favor or against of this norm. The ai people will obviously oppose construction of this norm but at least the focus is to shift it towards a direct dispute about what these norms should be. For instance the idea is Dario will have to say "I believe that these axioms should guide mathematics instead." This is what you want. It's better to search for an overshoot instead of to try to scramble together to negotiate.
You mention undergraduate performance dropping
> I'd like to just mention my own strong feelings/observation, clankers are only helping people with homework/doing it for them because it makes the clanker builders money and a consistent user base.
"it increasingly seems that artificial intelligence technologies are autonomously capable of doing the same"
> The obvious weakness of OAI's results are that
1. It's mightily suspicious that this run is claimed to have cost 1,000 API USD. That's getting close to free.
2. If 10 results, why not 100? 1,000,000? On team anthropic just having a clanker doing the cryptographic weakness was a big deal. Now two models that I suspected were neck-and-neck have totally different results.
3. If this product is in internal previews right now, finding 10 results seems quite fast. if more doesn't come quickly, i have reason to become suspicious of inconsistencies or issues thatr range from cherrypicking to something worse.
Do you have any thoughts here? If 10 open problems, then why not research how to build better clankers instead? Why not do more math on ai instead of this? This appears to imply the underlying strategics to me, which is that attention and flair is more important at this moment than driving more frontier LLM research?
"I believe that in a world where AI-led mathematics directly competes with human-led mathematics, AI-led mathematics will ultimately surpass us."
> Maybe. I read your link to Terence Tao's compiled ideas and he suggests that math can move and change over time. Supposedly we can't predict what work will be like. That being said, this is concerning.
"Perhaps the future career of the mathematician is that of a prompt engineer"
> Of course everything we're thinking of and panicking about in these timelines collapses any length of time into a brief moment. If it will kill mathematics, our concerns are now. If it will automate your job away or kill you one day, your concerns are now. If the mathematician is just a prompt engi, i'm implying this can't be true since the very first job that you would one day want to automate away if you're a company is prompt engineering, since you're just putting in a question. Clearly, the fact that there is leverage on the prompting side by the humans reflects either the reality or lack of innovation for companies to move there yet. This feels quite firm to me.
In the end this is a conceptually interesting proposition and it exists at the maximal edge between model capability and human capabilities. This is about clankers vs. humans, except with a sardonic twist. The humans are driving clanker development. When I say maximal, I think to myself that the only exception to this argument is if human cognition goes on to explore areas that LLMs just can't compete in one day and some weird job or means of sustainment and creation is invented.
I think your precision comes where you state that the world being created is just not one you want to live in.
Human ingenuity isn't dead yet. There are scaling laws and AI doesn't scale linearly. That is why OpenAI needs 1/2 a trillion dollars and a nuclear reactor to reach your level of intelligence. The real question is "when will AI develop good taste." That means "when will it pick good problems to solve" and not "when will it solve problems." Because right now it is hitting targets being held up by humans by emulating human created datasets. Once AI starts identifying research questions better than a human on its own that improve AI, working from datasets it created then we are all peasants. Right now it isn't clear that we can keep scaling AI much further and there may be a reprieve for our species.