The program and its guests

Y Combinator’s Paper Club (approximately 1 hour, 50 minutes and 48 seconds), hosted by Y Combinator, brings researchers and entrepreneurs, including members of the Together AI team and others, together to discuss inference. This article summarizes the core arguments of the discussion without attributing each statement to an individual speaker.

Key points

(The following paraphrases reflect the guests’ intended meaning; the timestamp links point to the corresponding locations in the original video.)

  • Inference is 12 times more important than training: taking DeepSeek V3, for which data is public, as an example, training used approximately 2.8 million H800 hours, while serving uses approximately 43 thousand H800 hours per day. For a single set of weights, annual inference compute is about 6 times training compute; over a 2-year lifespan, it is 12 times as much. The more users there are, the greater its share. (https://www.youtube.com/watch?v=mHNKuH-9OxU&t=101s）
  • Inference optimization has created approximately 220 billion US dollars in market value over the past 10 years: inference is more than calling a model; it is a full stack comprising routing, caching, scheduling, kernels, GPU memory and hardware, with trade-offs among speed, cost and quality at every step. (https://www.youtube.com/watch?v=mHNKuH-9OxU&t=61s）
  • The serving stack is the cost battleground: from request-level routing to silent changes across the entire serving stack, every layer affects the cost and latency per token. Companies that cannot run inference well will “lose their shirts” on cost of sales. (https://www.youtube.com/watch?v=mHNKuH-9OxU&t=627s）
  • Inference is becoming even more important: as AI becomes widespread and more users use the same set of weights, inference’s share of total compute will only continue to grow, and the relative weight of training and inference must be recalculated. (https://www.youtube.com/watch?v=mHNKuH-9OxU&t=131s）

Background and notes

The subtitles cover the full video (approximately 1 hour, 50 minutes and 48 seconds). This article summarizes the discussion’s core arguments; because the episode is a group discussion, the text presents the program’s arguments as a whole without attributing each statement to an individual speaker. Timestamps point to the relevant discussion segments.

Source links