This question is inspired by this Tim Gowers blogpost.
I have some familiarity with the work of Jacob Lurie, which contains ideas which are simply astounding; but what I don't understand is which key insight allowed him to begin his programme and achieve things which nobody had been able to achieve before. People had looked at $\infty$categories for years, and the idea of $(\infty,n)$categories is not in itself new. What was the key new idea which started "Higher Topos Theory", the proof of the BaezDolan cobordism hypothesis, "Derived Algebraic Geometry", etc.?

23$\begingroup$ There seems to be an implicit assumption in the above paragraph that Jacob had only one key idea, and I strongly disagree with that. I'd rather say that he has a rare combination of technical mastery and drive to view ideas in a broad context. $\endgroup$– S. Carnahan ♦Sep 6 '10 at 0:28

29$\begingroup$ I don't believe in superheros in math I believe in super ideas. I don't believe that Jacob Lurie was just trying the same thing everyone else was trying, and it happened to work for him because of greater intelligence or technical ability. There must have been key new ideas he brought to the table, and if Gromov is to be believed about the number of great ideas a mathematician has in life, then the number of these essentially new ideas is small. Most important, there's insight to be gained in knowing what these ideas are, because they led to breakthrough where so many had been stuck. $\endgroup$– Daniel MoskovichSep 6 '10 at 0:53

29$\begingroup$ Lots of math is done using a lot of good ideas put together, rather than a superidea. Believing in superideas seems to me just as naive as believing in superheroes. Certainly someone who works more efficiently or thinks more clearly would be able to get more good work done without needing to posit some special qualities of genius. What was Euler's superidea? (None of the above is meant to refer to Jacob's work specifically.) $\endgroup$– Noah SnyderSep 6 '10 at 3:45

26$\begingroup$ Can you, please, change the title to a more neutral form, perhaps, along the lines of "What are some key ideas behind higher topos theory"? Strange as this may sound, I think that using a personal name in the title is responsible for all the talk of superheros and other sillyness, which distracts from answering the question mathematically. $\endgroup$– Victor ProtsakSep 6 '10 at 9:31

12$\begingroup$ As an example of the kind of symbiosis that the internet makes possible so nicely, I shall now link from my blog post to this discussion. $\endgroup$– gowersSep 6 '10 at 10:39
My answer would be that his insight was firstly that it pays to take what Grothendieck said in his various long manuscripts, extremely seriously and then to devote a very large amount of thought, time and effort. Many of the methods in HTT have been available from the 1980s and the importance of quasicategories as a way to boost higher dimensional category theory was obvious to Boardman and Vogt even earlier. Lurie then put in an immense amount of work writing down in detail what the insights from that period looked like from a modern viewpoint.It worked as the progress since that time had provided tools ripe for making rapid progress on several linked problems. His work since HTT continues the momentum that he has built up.
As far as `abstracter than thou' goes, I believe that Grothendieck's ideas in Pursuing Stacks were not particularly abstract and Lurie's continuation of that trend is not either. Once you see that there are some good CONCRETE models for $\infty$categories the geometry involved gets quite concrete as well. Simplicial sets are not particularly abstract things, although they can be a bit scary when you meet them first. Quasicategories are then a simpleish generalisation of categories, but where you can use both categorical insights and homotopy insights. That builds a good intuition about infinity categories... now bring in modern algebraic topology with spectra, etc becoming available.


10$\begingroup$ Just a point of clarification: I am rather certain that he had not looked at any of Grothendieck's long manuscripts until well after his work on derived algebraic geometry had borne some fruit (in particular, after his infinity topoi preprint and his Ph.D.). This is of course assuming that by "long manuscripts" you did not mean published works like SGA. What you said in your answer does not strictly contradict what I'm saying, but I think it could be a source of confusion. $\endgroup$– S. Carnahan ♦Sep 6 '10 at 15:59

3$\begingroup$ That seems highly likely. The background ideas that from about 1984 onwards benefitted from some of the intuitions of, for instance, ` 'pursuing stacks' (PS) rarely used explicitly the constructions of that manuscript. They rather used the general context of those documents. AG sent PS to us in Bangor and Ronnie Brown helped in its distribution by providing (with explicit permission from AG) for copies to be sent to those people who requested it. It was discussed in seminars in various places with some very good mathematicians exploring some of the ideas from their own viewpoints. $\endgroup$ Sep 7 '10 at 5:59

2$\begingroup$ (continued) This also happened with others of the lengthy manuscripts he wrote at about that time, although without Bangor's involvement (so I am not sure of the mechanisms). AG's esquisse d'une programme is another document that is very influential. This process of course is not restricted to this context but is one of the many ways maths grows through its transmission. (If you do not know the story of our involvement in PS you may like to look at Ronnie's webpage: bangor.ac.uk/~mas010/pstacks.htm where he tells the story from his own point of view.) $\endgroup$ Sep 7 '10 at 6:07

$\begingroup$ Just to update on Ronnie Brown's web page. The Bangor page was shut down by the university. Ronnie transferred his page on pursuing stacks to: groupoids.org.uk/pstacks.html $\endgroup$ Jan 15 '18 at 16:19
I think, one of the key insights underlying derived algebraic geometry and Lurie's treatment of elliptic cohomology is taking some ideas of Grothendieck really serious. Two manifestations:
1) One of the points of the scheme theory initiated by Grothendieck is the following: if one takes intersection of two varieties just on the pointset level, one loses information. One has to add the possibility of nilpotents (somewhat higher information) to preserve the information of intersection multiplicities and get the "right" notion of a fiber product. Now one of the points of derived algebraic geometry (as explained very lucidly in the introduction to DAG V) is that for homological purposes this is not really the right fiber product  you need to take some kind of homotopy fiber product. This is, because one still loses information because one is taking quotients  one should add isomorphism instead and view it on a categorical level. Thus, you can take a meaningful intersection of a point with itself, for example. This is perhaps an instance where the homological revolution, which went to pure mathematics last century, benefits from a second wave, a homotopical revolution  if I am allowed to overstate this a bit.
2) Another insight of Grothendieck and his school was, how important it is to represent functors in algebraic geometry  regardless of what you want at the end. [as Mazur reports, Hendrik Lenstra was once sure that he did want to solve Diophantine equations and did not want to represent functors  and later he was amused that he represented functors to solve Diophantine equations.] And this is Lurie's approach to elliptic cohomology and tmf: Hopkins and Miller showed the existence of a certain sheaf of $E_\infty$ring spectra on the moduli stack of elliptic curves. Lurie showed that this represents a derived moduli problem (of oriented derived elliptic curves).
Also his solution of the cobordism hypothesis has a certain flavor of Grothendieck: you have to put things in a quite general framework to see the essence. This philosophy also shines quite clearly through his DAG, I think.
Besides, I do not think, there is a single key insight in Higher Topos Theory besides the feeling that infinitycategories are important and that you can find analogues to most of classical category theory in quasicategories. Then there are lots of little (but every single one amazing) insights, how this transformation from classical to infinitycategory theory works.

1$\begingroup$ Of course one suspects that selectivity also has a role to play, in dealing with "big abstraction". "Abstracter than thou" is not in itself a potent heuristic, despite the success Grothendieck had with it: these days I suppose one also wonders what is the catalyst when big abstract machines are cranked up. There is still something arcane about that, it seems. $\endgroup$ Sep 6 '10 at 10:58

2$\begingroup$ Rather than "abstracter than thou", Jacob has, I think, a specific fixed abstract framework through whose lens he views mathematics. I don't understand, though, which specific quality of this particular abstract framework makes it better than all others i.e. where exactly the conceptual breakthrough occured. This response partially answers. $\endgroup$ Sep 6 '10 at 11:06
One of the things that Jacob Lurie finds important is to "think invariantly": Do not use model dependent definitions. Do not prove modeldependent results.
Here's an example to illustrate what I mean:
"The $\infty$category of spetra is the free stable $\infty$category with colimits generated by a single object".
That's a nice model independent definition of the $\infty$category of spectra.
I think one of the insights leading to his successes, which is of course not unique to him, is that when doing higher category theory it is useful to add invertible higher cells going all the way up to the top before you try to add noninvertible ones anywhere. This is along the same lines as Noah's answer: the cobordism hypothesis, which was originally stated in terms of ncategories, becomes easier once you generalize it to a stronger statement about (∞,n)categories, since then induction becomes easier/possible.
My understanding (which isn't very deep and so someone should please correct me if I'm wrong) is that one key idea in the HopkinsLurie proof of the cobordism hypothesis was to generalize the cobordism hypothesis to allow more general types of cobordism categories. This stronger result the becomes easier to prove because they could do a doubleinduction argument that moves between the (stably n)framed case (which is the usual cobordism hypothesis) and the oriented case. Furthermore, it means that the final results they get are stronger, because you get a classification not just of framed TFTs, but of TFTs with more general kinds of structures on the bordism categories.
Daniel,
have you tried asking Jacob?

12$\begingroup$ Since Jacob is on MO, I am secretly hoping for him to notice this question so I can add another example to tea.mathoverflow.net/discussion/516/… $\endgroup$ Oct 3 '10 at 19:58

2$\begingroup$ I have a feeling that this is not a particularly useful answer for people who have no opportunity to ask Lurie $\endgroup$– user74900May 15 '18 at 22:04

$\begingroup$ @AknazarKazhymurat: Who doesn't have an opportunity? $\endgroup$ May 17 '18 at 0:13

$\begingroup$ @SandorKovacs I reckon he does not reply to emails from some random mathematicians; maybe I am wrong though. Of course everyone has an opportunity to ask, but to expect your question to be answered is a different matter I think $\endgroup$– user74900May 17 '18 at 10:54

$\begingroup$ Right. But, I would say that what I said amounts to this. Daniel could try to ask Jacob and he might answer. If Daniel has tried and have not received an answer then we can theorize, but the ultimate answer would have to come from Jacob one way or another. You may not feel like asking him, but I stand by the claim that that's the best, and in fact essentially the only way to get a correct answer to this question. Everything else is guessing. By the way as Willie Wong points out Jacob Lurie might actually sees this post and answers. One could even ping him, so he would look at it... $\endgroup$ May 18 '18 at 0:12