OneRec: Unifying Retrieve and Rank with Generative Recommender and Preference Alignment

Jiaxin Deng KuaiShou Inc.Beijing, China dengjiaxin03@kuaishou.com, Shiyao Wang KuaiShou Inc.Beijing, China wangshiyao08@kuaishou.com, Kuo Cai KuaiShou Inc.Beijing, China caikuo@kuaishou.com, Lejian Ren KuaiShou Inc.Beijing, China renlejian@kuaishou.com, Qigen Hu KuaiShou Inc.Beijing, China huqigen03@kuaishou.com, Weifeng Ding KuaiShou Inc.Beijing, China dingweifeng@kuaishou.com, Qiang Luo KuaiShou Inc.Beijing, China luoqiang@kuaishou.com and Guorui Zhou KuaiShou Inc.Beijing, China zhouguorui@kuaishou.com

(2018)

Abstract.

Recently, generative retrieval-based recommendation systems (GRs) have emerged as a promising paradigm by directly generating candidate videos in an autoregressive manner. However, most modern recommender systems adopt a retrieve-and-rank strategy, where the generative model functions only as a selector during the retrieval stage. In this paper, we propose OneRec, which replaces the cascaded learning framework with a unified generative model. To the best of our knowledge, this is the first end-to-end generative model that significantly surpasses current complex and well-designed recommender systems in real-world scenarios. Specifically, OneRec includes: 1) an encoder-decoder structure, which encodes the user’s historical behavior sequences and gradually decodes the videos that the user may be interested in. We adopt sparse Mixture-of-Experts (MoE) to scale model capacity without proportionally increasing computational FLOPs. 2) a session-wise generation approach. In contrast to traditional next-item prediction, we propose a session-wise generation, which is more elegant and contextually coherent than point-by-point generation that relies on hand-crafted rules to properly combine the generated results. 3) an Iterative Preference Alignment module combined with Direct Preference Optimization (DPO) to enhance the quality of the generated results. Unlike DPO in NLP, a recommendation system typically has only one opportunity to display results for each user’s browsing request, making it impossible to obtain positive and negative samples simultaneously. To address this limitation, We design a reward model to simulate user generation and customize the sampling strategy according to the attributes of the recommendation system’s online learning. Extensive experiments have demonstrated that a limited number of DPO samples can align user interest preferences and significantly improve the quality of generated results. We deployed OneRec in the main scene of Kuaishou, a short video recommendation platform with hundreds of millions of daily active users, achieving a 1.6% increase in watch-time, which is a substantial improvement.

Generative Recommendation, Autoregressive Generation, Semantic Tokenization, Direct Preference Optimization

1. Introduction

To balance efficiency and effectiveness, most modern recommender systems adopt a cascade ranking strategy ^{1 2 3 4}. As illustrated in Figure 1(b), a typical cascade ranking system employs a three-stage pipeline: recall ^{1 5 6}, pre-ranking ^{7 8}, and ranking ^{9 10 11 12 13 14 15}. Each stage is responsible for selecting the top- $k$ items from the received items and passing the results to the next stage, collectively balancing the trade-off between system response time and sorting accuracy.

Figure 1. (a) Our proposed unified architecture for end-to-end generation. (b) A typical cascade ranking system, which includes three stages from the bottom to the top: Retrieval, Pre-ranking, and Ranking.

Although efficient in practice, existing methods typically treat each ranker independently, where the effectiveness of each isolated stage serves as the upper bound for the subsequent ranking stage, thereby limiting the performance of the overall ranking system. Despite various efforts ^{16 17 18 4 19 20} to improve overall recommendation performance by enabling interaction among rankers, they still maintain the traditional cascade ranking paradigm. Recently, generative retrieval-based recommendation systems (GRs) ^{21 22 23} have emerged as a promising paradigm by directly generating the identifier of a candidate item in an autoregressive sequence generation manner. By indexing items with quantized semantic IDs that encode item semantics ²⁴, recommenders can leverage the abundant semantic information within the items. The generative nature of GRs makes them suitable for directly selecting candidate items through beam search decoding and producing more diverse recommendation results. However, current generative models only act as selectors in the retrieval stage, as their recommendation accuracy does not yet match that of well-designed multiple cascade rankers.

To address the above challenges, we propose a unified end-to-end generative framework for single-stage recommendation named OneRec. First, we present an encoder-decoder architecture. Taking inspiration from the scaling laws observed in training large language models, we find that scaling the capacity of recommendation models also consistently improves the performance. So we scale up the model parameters based on the structure of MoE ^{25 26 27}, which significantly improves the model’s ability to characterize user interests. Second, unlike the traditional point-by-point prediction of the next item, we propose a session-wise list generation approach that considers the relative content and order of the items within each session. The point-by-point generation method necessitates hand-craft strategies to ensure coherence and diversity in the generated results. In contrast, the session-wise learning process enables the model to autonomously learn the optimal session structure by feeding it preferred data. Last but not least, we explore preference learning by using direct preference optimization (DPO) ²⁸ to further enhance the quality of the generated results. For constructing preference pairs, we take inspiration from hard negative sampling ²⁹ by creating self-hard rejected samples from the beam search results rather than random sampling. We propose an Iterative Preference Alignment (IPA) strategy to rank the sampled responses based on scores provided by the pre-trained reward model (RM), identifying the best-chosen and worst-rejected samples. Our experiments on large-scale industry datasets show the superiority of the proposed method. We also conduct a series of ablation experiments to demonstrate the effectiveness of each module in detail. The main contributions of this work are summarized as follows:

• To overcome the limitations of cascade ranking, we introduce OneRec, a single-stage generative recommendation framework. To the best of our knowledge, this is one of the first industrial solutions capable of handling item recommendations with a unified generation model, significantly surpassing the traditional multi-stage ranking pipeline.
• We highlight the necessity of model capacity and contextual information of target items through a session-wise generation manner, which enables more accurate predictions and enhances the diversity of generated items.
• We propose a novel self-hard negative samples selection strategy based on personalized reward model. With direct preference optimization, we enhance OneRec’s generalization across a broader range of user preference. Extensive offline experiments and online A/B testing demonstrates their effectiveness and efficiency.

Figure 2. The overall framework of OneRec, consists of two stages: (i) the session training stage which train OneRec with session-wise data; (ii) the IPA stage which utilizes iterative direct preference optimization with self-hard negatives.

2. Related Work

2.1. Generative Recommendation

In recent years, with the remarkable progress in generative models, generative recommendation has received increasing attention. Unlike traditional embedding-based retrieval methods which largely rely on a two-tower model for calculating the ranking score for each candidate item and utilize an effecient MIPS or ANN ^{30 31 32 33 34} search system for retrieving top- $k$ relevant items. Generative Retrieval (GR) ³⁵ method formulates the problem of retrieving relevant documents from the database as a sequence generation task which generate the relevant document tokens sequentially. The document tokens can be the document titles, document IDs or pre-trained semantic IDs ³⁶. GENRE ³⁷ first adopts the transformer architecture for entity retrieval, generating entity names in an autoregressive fashion based on the conditioned context. DSI ³⁶ first proposes the concept of assigning structured semantic IDs to documents and training encoder-decoder models for generative document retrieval. Following this paradigm, TIGER ²¹ introduces the formulation of generative item retrieval models for recommender systems.

In addition to the generation framework, how to index items has also attracted increasing attention. Recent research focuses on the semantic indexing technique ^{21 36 38}, which aims to index items based on content information. Specifically, TIGER ²¹ and LC-Rec ²³ apply residual quantization (RQ-VAE) to textual embeddings derived from item titles and descriptions for tokenization. Recforest ³⁸ utilizes hierarchical k-means clustering on item textual embeddings to obtain cluster indexes as tokens. Furthermore, recent studies such as EAGER ²² explore integrating both semantic and collaborative information into the tokenization process.

2.2. Preference Alignment of Language Models

During the post-training ³⁹ phase of Large Language Models (LLMs), Reinforcement Learning from Human Feedback (RLHF) ^{40 41} is a prevalent method in aligning LLMs with human values by employing reinforcement learning techniques guided by reward models that represent human feedback. However, RLHF suffers from instability and inefficiency. Direct Preference Optimization (DPO) ²⁸ is proposed which derives the optimal policy in closed form and enables direct optimization using preference data. Apart from that, several variants have been proposed to further improve the original DPO. For example, IPO ⁴² bypasses two approximations in DPO with a general objective. cDPO ²⁸ alleviates the influence of noisy labels by introducing a hyperparameter $ϵ$. rDPO ⁴³ designs an unbiased estimate of the original Binary Cross Entropy loss. Other variants including CPO ⁴⁴, simDPO ⁴³, also enhance or expand DPO in various aspects. However, unlike traditional NLP scenarios where preference data is explicitly annotated through humans, preference learning in recommendation systems faces a unique challenge because of the sparsity of user-item interaction data. This challenge results in adapting DPO for recommendation are still largely unexplored. Different from S-DPO which focuses on incorporating multiple negatives in user preference data for LM-based recommenders, we train a reward model and based on the scores from reward model we choose personalized preference data for different users.

3. Methods

In this section, we propose OneRec, an end-to-end framework that generates target items through a single-stage retrieval manner. In Section 3.1, we first introduce the feature engineering for the single-stage generative recommendation pipeline in industrial applications. Then, in Section 3.2, we formally define the session-wise generative tasks and present the architecture of our proposed OneRec model. Finally, we elaborate on the model’s capability with a personalized reward model for self-hard negative sampling in Section 3.3, and demonstrate how we iteratively improve model performance through direct preference optimization. The overall framework of OneRec is illustrated in Figure 2.

3.1. Preliminary

In this section, we introduce the construction of the single-stage generative recommendation pipeline from the perspectives of feature engineering. For user-side feature, OneRec takes the positive historical behavior sequences ${\mathcal{H}}_u=\{{\mathbf{v}}_1^h,{\mathbf{v}}_2^h,\dots ,{\mathbf{v}}_n^h\}$ as input, where $\mathbf{v}$ represent the videos that the user has effectively watched or interacted with (likes, follows, shares), and $n$ is the length of behaviour sequence. The output of OneRec is a list of videos, consisting of a session $\mathcal{S}=\{{\mathbf{v}}_1,{\mathbf{v}}_2,...,{\mathbf{v}}_m\}$, where $m$ is the number of videos within a session (the detailed definition of “session” can be found in Section 3.2).

For each video ${\mathbf{v}}_i$, we describe them with multi-modal embeddings ${\text{e}}_i\in {\mathbb{R}}^d$ which are aligned with the real user-item behaviour distribution ⁴⁵. Based on the pretrain multi-modal representation, existing generative recommendation frameworks ^{46 21} use RQ-VAE ⁴⁷ to encode the embedding into semantic tokens. However, such method is suboptimal due to the unbalanced code distribution which is known as the hourglass phenomenon ⁴⁸. We apply a multi-level balanced quantitative mechanism to transform the ${\text{e}}_i$ with residual K-Means quantization algorithm ⁴⁵. At the first level ($l=1$), the initial residual is defined as ${\mathbf{r}}_i^1={\mathbf{e}}_i$. At each level $l$, we have a codebook ${\mathcal{C}}_l=\{{\mathbf{c}}_1^l,...,{\mathbf{c}}_K^l\}$, where $K$ is the codebook size. The index of the closest centroid node embedding is generate through ${\mathbf{s}}_i^l=\arg {\min }_k\Vert {\text{r}}_i^l-{\mathbf{c}}_k^l{\Vert }_2^2$ and for next level $l+1$ the residual is defined as ${\mathbf{r}}_i^{l+1}={\mathbf{r}}_i^l-{\mathbf{c}}_{{\mathbf{s}}_i^l}^l$. So the corresponding codebook tokens are generated through hierarchical indexing:

$${\mathbf{s}}_i^1=\arg \underset{k}{\min }\Vert {\mathbf{r}}_i^1-{\mathbf{c}}_k^1{\Vert }_2^2,{\mathbf{r}}_i^2={\mathbf{r}}_i^1-{\mathbf{c}}_{{\mathbf{s}}_i^1}^1$$ $${\mathbf{s}}_i^2=\arg \underset{k}{\min }\Vert {\mathbf{r}}_i^2-{\mathbf{c}}_k^2{\Vert }_2^2,{\mathbf{r}}_i^3={\mathbf{r}}_i^2-{\mathbf{c}}_{{\mathbf{s}}_i^2}^2$$ $$⋮$$ $${\mathbf{s}}_i^L=\arg \underset{k}{\min }\Vert {\mathbf{r}}_i^L-{\mathbf{c}}_k^L{\Vert }_2^2$$

where $L$ is the total layers of sematic ID.

To construct a balanced codebook ${\mathcal{C}}_l=\{{\mathbf{c}}_1^l,\dots ,{\mathbf{c}}_K^l\}$, we apply the Balanced K-means as detailed in Algorithm 1 for itemset partitioning. Given the total video set $\mathcal{V}$, this algorithm partitions the set into $K$ clusters, where each cluster contains exactly $w=|\mathcal{V}|/K$ videos. During iterative computation, each centroid is sequentially assigned its $w$ nearest unallocated videos based on Euclidean distance, followed by centroid recalibration using mean vectors of assigned videos. The termination criterion is satisfied when cluster assignments reach convergence.

Input: Item set $\mathcal{V}$, number of clusters $K$

Compute $w\leftarrow |\mathcal{V}|/K$

Initialize centroids ${\mathcal{C}}_l=\{{\mathbf{c}}_1^l,\dots ,{\mathbf{c}}_K^l\}$ with random selection;

repeat

Initialize unassigned set $\mathcal{U}\leftarrow \mathcal{V}$

for each cluster $k\in \{1,\dots ,K\}$ do

Sort $\mathcal{U}$ by ascending distance from centroid ${\mathbf{c}}_k^l$;

Assign ${\mathcal{V}}_k\leftarrow \mathcal{U}[0:w-1]$;

Update centroid ${\mathbf{c}}_k^l\leftarrow \frac{1}{w}{\sum }_{{\mathbf{r}}^l\in {\mathcal{V}}_k}{\mathbf{r}}^l$;

Remove assigned items $\mathcal{U}\leftarrow \mathcal{U}\setminus {\mathcal{V}}_k$;

end for

until Assignment convergence;

Output: Optimized codebook ${\mathcal{C}}_l=\{{\mathbf{c}}_1^l,\dots ,{\mathbf{c}}_K^l\}$

Algorithm 1 Balanced K-means Clustering

3.2. Session-wise List Generation

Different from traditional point-wise recommendation methods that only predict the next video, session-wise generation aims to generate a list of high-value sessions based on users’ historical interaction sequences, which enables the recommendation model to capture the dependencies between videos in the recommended list. Specifically, a session refers to a batch of short videos returned in response to a user’s request, typically consisting of 5 to 10 videos. The videos within a session generally take into account factors such as user interest, coherence, and diversity. We have devised several criteria to identify high-quality sessions, including:

• The number of short videos actually watched by the user within a session is greater than or equal to 5;
• The total duration for which the user watches the session exceeds a certain threshold;
• The user exhibits interaction behaviors, such as liking, collecting, or sharing the videos;

This approach ensures that our session-wise model learns from real user engagement patterns and captures more accurate contextual information within the session list. So the objective of our session-wise model $\mathcal{M}$ can be formalized as:

$$\mathcal{S}:=\mathcal{M}({\mathcal{H}}_u)$$

where ${\mathcal{H}}_u$ is represented from the remantic IDs: $$\begin{gathered} {\mathcal{H}}_u=\{({\mathbf{s}}_1^1,{\mathbf{s}}_1^2,\cdots , \\ {\mathbf{s}}_1^L),({\mathbf{s}}_2^1,{\mathbf{s}}_2^2,\cdots ,{\mathbf{s}}_2^L),\cdots ,({\mathbf{s}}_n^1,{\mathbf{s}}_n^2,\cdots ,{\mathbf{s}}_n^L)\} \end{gathered}$$ and $$\begin{gathered} \mathcal{S}=\{({\mathbf{s}}_1^1,{\mathbf{s}}_1^2,\cdots ,{\mathbf{s}}_1^L), \\ ({\mathbf{s}}_2^1,{\mathbf{s}}_2^2,\cdots ,{\mathbf{s}}_2^L),\cdots ,({\mathbf{s}}_m^1,{\mathbf{s}}_m^2,\cdots ,{\mathbf{s}}_m^L)\} \end{gathered}$$.

As shown in Figure 2 (a), consistent with the T5 ⁴⁹ architecture, our model employs a transformer-based framework consisting of two main components: an encoder for modeling user historical interactions and a decoder for session list generation. Specifically, the encoder leverages the stacked multi-head self-attention and feed-forward layers to process the input sequence ${\mathcal{H}}_u$. We denote the encoded historical interaction features as $\text{H}=Encoder({\mathcal{H}}_u)$.

The decoder takes the semantic IDs of the target session as input and generates the target in an auto-regressive manner. To train a larger model at reasonable economic costs, for the feed-forward neural networks (FNNs) in the decoder, we adopt the MoE architecture ^{25 26 27} commonly used in Transformer-based language models and substitute the $l$ -th FFN with:

$$\begin{array}{rl}{\mathbf{H}}_t^{l+1}& =\sum _{i=1}^{N_{\mathrm{MoE}}}\left(g_{i,t}{\mathrm{FFN}}_i\left({\mathbf{H}}_t^l\right)\right)+{\mathbf{H}}_t^l,\\ g_{i,t}& =\{\begin{array}{ll}{s}_{i,t},& s_{i,t}\in \mathrm{Topk}(\{s_{j,t}|1\le j\le N\},K_{\mathrm{MoE}}),\\ 0,& \text{otherwise},\end{array}\\ s_{i,t}& ={\mathrm{Softmax}}_i\left({{\mathbf{H}}_t^l}^T{\mathbf{e}}_i^l\right),\end{array}$$

where $N_{\mathrm{MoE}}$ represents the total number of experts, ${\mathrm{FFN}}_i(\cdot )$ is the $i$ -th expert FFN, and $g_{i,t}$ denotes the gate value for the $i$ -th expert. The gate value $g_{i,t}$ is sparse, meaning that only $K_{\mathrm{MoE}}$ out of $N_{\mathrm{MoE}}$ gate values are non-zero. This sparsity property ensures computational efficiency within an MoE layer and each token will be assigned to and computed in only $K_{\mathrm{MoE}}$ experts.

For training, we add a start token ${\mathbf{s}}_{[\mathrm{BOS}]}$ at the beginning of codes to construct the decoder inputs with:

$$\begin{array}{c}\bar{\mathcal{S}}=\{{\mathbf{s}}_{[\mathrm{BOS}]},{\mathbf{s}}_1^1,{\mathbf{s}}_1^2,\cdots ,{\mathbf{s}}_1^L,{\mathbf{s}}_{[\mathrm{BOS}]},{\mathbf{s}}_2^1,{\mathbf{s}}_2^2,\cdots ,{\mathbf{s}}_2^L,\\ \cdots ,{\mathbf{s}}_{[\mathrm{BOS}]},{\mathbf{s}}_m^1,{\mathbf{s}}_m^2,\cdots ,{\mathbf{s}}_m^L\}\end{array}$$

We utilize cross-entropy loss for next-token prediction on the sematic IDs of the target session. The NTP loss ${\mathcal{L}}_{\mathrm{NTP}}$ is formulated as:

$$\begin{array}{c}{\mathcal{L}}_{\mathrm{NTP}}=-\sum _{i=1}^m\sum _{j=1}^L\log P({\mathbf{s}}_i^{j+1}\mid [{\mathbf{s}}_{[\mathrm{BOS}]},{\mathbf{s}}_1^1,{\mathbf{s}}_1^2,\cdots ,{\mathbf{s}}_1^L,\cdots ,\\ {\mathbf{s}}_{[\mathrm{BOS}]},{\mathbf{s}}_i^1,\cdots ,{\mathbf{s}}_i^j];\Theta ).\end{array}$$

After a certain amount of training on session-wise list generation task, we obtain the seed model ${\mathcal{M}}_t$.

3.3. Iterative Preference Alignment with RM

The high-quality sessions defined in Section 3.2 provide valuable training data, enabling the model to learn what constitutes a good session, thereby ensuring the quality of generated videos. Building on this foundation, we aim to further enhance the model’s ability by Direct Preference Optimization (DPO). In traditional natural language processing (NLP) scenarios, preference data is explicitly annotated by humans. However, preference learning in recommendation systems confronts a unique challenge due to the sparsity of user-item interaction data, which necessitates a reward model (RM). So we introduce a session-wise reward model in Section 3.3.1. Moreover, we improve the conventional DPO by proposing an iterative direct preference optimization that enables the model to self-improvement described in Section 3.3.2.

for $epoch\leftarrow t$ to $T$ do

for $sample\leftarrow 1$ to $N_{\mathrm{sample}}$ do

if $rand()<r_{\mathrm{DPO}}$ then

for $i\leftarrow 1$ to $N$ do

${\mathcal{S}}_u^i\sim {\mathcal{M}}_t({\mathcal{H}}_u)$;

$r_u^i\leftarrow R(\mathbf{u},{\mathcal{S}}_u^i)$;

end for

${\mathcal{S}}_u^w\leftarrow {\mathcal{S}}_u^{\arg {\max }_i{r}_u^i}$;

${\mathcal{S}}_u^l\leftarrow {\mathcal{S}}_u^{\arg {\min }_i{r}_u^i}$;

Compute NTP and DPO loss:

$\mathcal{L}\leftarrow {\mathcal{L}}_{\mathrm{NTP}}+\lambda {\mathcal{L}}_{\mathrm{DPO}}$;

else

Compute NTP loss:

$\mathcal{L}\leftarrow {\mathcal{L}}_{\mathrm{NTP}}$;

end if

Update parameters:

$\Theta \leftarrow \Theta -\alpha {\nabla }_{\Theta }\mathcal{L}$;

end for

Update model snapshot: ${\mathcal{M}}_{t+1}\leftarrow {\mathcal{M}}_t$;

end for

Output: Optimized parameters $\Theta$

Algorithm 2 Iterative Preference Alignment (IPA)

3.3.1. Reward Model Training

We use $R(\mathbf{u},\mathcal{S})$ to denote the reward model which selects preference data for different users. Here, the output $r$ represents the reward corresponding to user $u$ ’s (usually represented by user behavior) preference on the session $\mathcal{S}=\{{\mathbf{v}}_1,{\mathbf{v}}_2,\dots ,{\mathbf{v}}_m\}$. In order to equip the RM with the capacity to rank session, we first extract the target-aware representation ${\mathbf{e}}_i={\mathbf{v}}_i\odot \mathbf{u}$ of each item ${\mathbf{v}}_i$ in $\mathcal{S}$, where $\odot$ represents the target-aware operation (such as target attention toward user behavior). So we get the target-aware representation $\mathbf{h}=\{{\mathbf{e}}_1,{\mathbf{e}}_2,\cdots ,{\mathbf{e}}_m\}$ for session $\mathcal{S}$. Then the items within a session interact with each other through self-attention layers to fuse the necessary information among different items:

$${\mathbf{h}}_f=\mathrm{SelfAttention}(\mathbf{h}{\mathbf{W}}_s^Q,\mathbf{h}{\mathbf{W}}_s^K,\mathbf{h}{\mathbf{W}}_s^V)$$

Next we utilize different tower to make predictions on multi-target reward and the RM is pre-trained with abundant recommendation data:

\small\begin{split}\hat{r}^{swt}&=\texttt{Tower}^{swt}\big{(}\texttt{Sum}\big{% (}\bm{h}_{f}\big{)}\big{)},\hat{r}^{vtr}=\texttt{Tower}^{vtr}\big{(}\texttt{% Sum}\big{(}\bm{h}_{f}\big{)}\big{)},\\ \hat{r}^{wtr}&=\texttt{Tower}^{wtr}\big{(}\texttt{Sum}\big{(}\bm{h}_{f}\big{)}% \big{)},\hat{r}^{ltr}=\texttt{Tower}^{ltr}\big{(}\texttt{Sum}\big{(}\bm{h}_{f}% \big{)}\big{)},\\ &\texttt{whe}\texttt{re}\quad\texttt{Tower}(\cdot)=\texttt{Sigmoid}\big{(}% \texttt{MLP}(\cdot)\big{)}\end{split}

After getting all the estimated rewards ${\hat{r}}^{swt},\dots$ and the ground-truth labels $y^{swt},\dots$ for each session, we directly minimize the binary cross-entropy loss to train the RM. The loss function ${\mathcal{L}}_{\mathrm{RM}}$ is defined as follows:

$$\begin{array}{r}{\mathcal{L}}_{\mathrm{RM}}=-\sum _{swt,\dots }^{xtr}\left(y^{xtr}\log ({\hat{r}}^{xtr})+(1-y^{xtr})\log (1-{\hat{r}}^{xtr})\right)\end{array}$$

3.3.2. Iterative Preference Alignment

Based on pre-trained RM $R(\mathbf{u},\mathcal{S})$ and current OneRec ${\mathcal{M}}_t$, we generate $N$ different responses for each user by beam search:

$${\mathcal{S}}_u^n\sim M_t({\mathcal{H}}_u)\text{for all}u\in \mathcal{U}\text{and}n\in [N],$$

where we use $[N]$ to denote $\{1,2,\dots ,N\}$.

Then we computes the reward $r_u^n$ for each of these responses based on the RM $R(\mathbf{u},\mathcal{S})$:

$$r_u^n=R(\mathbf{u},{\mathcal{S}}_u^n)$$

Next we build the preference pairs $D_t^{\text{pairs}}=({\mathcal{S}}_u^w,{\mathcal{S}}_u^l,{\mathcal{H}}_u)$ by choosing the winner response $({\mathcal{S}}_u^w,{\mathcal{H}}_u)$ with the highest reward value and a loser response $({\mathcal{S}}_u^l,{\mathcal{H}}_u)$ with the lowest reward value. Given the preference pairs, we can now train a new model $M_{t+1}$ which is initialized from model $M_t$, and updated with a loss function that combines the DPO loss ²⁸ for learning from the preference pairs. The loss corresponding to each preference pair is as follows:

$$\begin{array}{rl}{\mathcal{L}}_{\text{DPO}}& ={\mathcal{L}}_{\text{DPO}}({\mathcal{S}}_u^w,{\mathcal{S}}_u^l|{\mathcal{H}}_u)\\ & =-\log \sigma \left(\beta \log \frac{M_{t+1}({\mathcal{S}}_u^w|{\mathcal{H}}_u)}{M_t({\mathcal{S}}_u^w|{\mathcal{H}}_u)}-\beta \log \frac{M_{t+1}({\mathcal{S}}_u^l|{\mathcal{H}}_u)}{M_t({\mathcal{S}}_u^l|{\mathcal{H}}_u)}\right).\end{array}$$

As shown in Algorithm 2 and Figure 2 (b), the overall procedure involves training a sequence of models $M_t,\dots ,M_T$. To mitigate the computational burden during beam search inference, we randomly sample only $r_{\mathrm{DPO}}=1\%$ data for preference alignment. For each successive model $M_{t+1}$, it initializes from previous model $M_t$ and utilizes the preference data $D_t^{\text{pairs}}$ generated by the $M_t$ for training.

Table 1. Offline performance of our proposed OneRec (green) with pointwise methods (brown), listwise methods (blue) and preference alignment methods (yellow). Best results are in bold, sub-optimal results are underlined. Metrics with ↑ \uparrow indicate higher is better, while ↓ \downarrow indicates lower is better.

Footnotes

1. Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems. 191–198. ↩ ↩²

2. Shichen Liu, Fei Xiao, Wenwu Ou, and Luo Si. 2017. Cascade ranking for operational e-commerce search. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1557–1565. ↩

3. Lidan Wang, Jimmy Lin, and Donald Metzler. 2011. A cascade ranking model for efficient ranked retrieval. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval. 105–114. ↩

4. Jiarui Qin, Jiachen Zhu, Bo Chen, Zhirong Liu, Weiwen Liu, Ruiming Tang, Rui Zhang, Yong Yu, and Weinan Zhang. 2022. Rankflow: Joint optimization of multi-stage cascade ranking systems as flows. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 814–824. ↩ ↩²

5. Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management. 2333–2338. ↩

6. Han Zhu, Xiang Li, Pengye Zhang, Guozheng Li, Jie He, Han Li, and Kun Gai. 2018. Learning tree-based deep model for recommender systems. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. 1079–1088. ↩

7. Xu Ma, Pengjie Wang, Hui Zhao, Shaoguo Liu, Chuhan Zhao, Wei Lin, Kuang-Chih Lee, Jian Xu, and Bo Zheng. 2021. Towards a better tradeoff between effectiveness and efficiency in pre-ranking: A learnable feature selection based approach. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2036–2040. ↩

8. Zhe Wang, Liqin Zhao, Biye Jiang, Guorui Zhou, Xiaoqiang Zhu, and Kun Gai. 2020. Cold: Towards the next generation of pre-ranking system. arXiv preprint arXiv:2007.16122 (2020). ↩

9. Christopher JC Burges. 2010. From ranknet to lambdarank to lambdamart: An overview. Learning 11, 23-581 (2010), 81. ↩

10. Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. DeepFM: a factorization-machine based neural network for CTR prediction. arXiv preprint arXiv:1703.04247 (2017). ↩

11. B Hidasi. 2015. Session-based Recommendations with Recurrent Neural Networks. arXiv preprint arXiv:1511.06939 (2015). ↩

12. Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. 2019. Deep interest evolution network for click-through rate prediction. In Proceedings of the AAAI conference on artificial intelligence, Vol. 33. 5941–5948. ↩

13. Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. 1059–1068. ↩

14. Qi Pi, Guorui Zhou, Yujing Zhang, Zhe Wang, Lejian Ren, Ying Fan, Xiaoqiang Zhu, and Kun Gai. 2020. Search-based user interest modeling with lifelong sequential behavior data for click-through rate prediction. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 2685–2692. ↩

15. Jianxin Chang, Chenbin Zhang, Zhiyi Fu, Xiaoxue Zang, Lin Guan, Jing Lu, Yiqun Hui, Dewei Leng, Yanan Niu, Yang Song, et al. 2023. TWIN: TWo-stage interest network for lifelong user behavior modeling in CTR prediction at kuaishou. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 3785–3794. ↩

16. Luke Gallagher, Ruey-Cheng Chen, Roi Blanco, and J Shane Culpepper. 2019. Joint optimization of cascade ranking models. In Proceedings of the twelfth ACM international conference on web search and data mining. 15–23. ↩

17. Hongliang Fei, Jingyuan Zhang, Xingxuan Zhou, Junhao Zhao, Xinyang Qi, and Ping Li. 2021. GemNN: gating-enhanced multi-task neural networks with feature interaction learning for CTR prediction. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval. 2166–2171. ↩

18. Jiri Hron, Karl Krauth, Michael Jordan, and Niki Kilbertus. 2021. On component interactions in two-stage recommender systems. Advances in neural information processing systems 34 (2021), 2744–2757. ↩

19. Xu Huang, Defu Lian, Jin Chen, Liu Zheng, Xing Xie, and Enhong Chen. 2023. Cooperative Retriever and Ranker in Deep Recommenders. In Proceedings of the ACM Web Conference 2023. 1150–1161. ↩

20. Yunli Wang, Zhiqiang Wang, Jian Yang, Shiyang Wen, Dongying Kong, Han Li, and Kun Gai. 2024a. Adaptive Neural Ranking Framework: Toward Maximized Business Goal for Cascade Ranking Systems. In Proceedings of the ACM on Web Conference 2024. 3798–3809. ↩

21. Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan Hulikal Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Tran, Jonah Samost, et al. 2023. Recommender systems with generative retrieval. Advances in Neural Information Processing Systems 36 (2023), 10299–10315. ↩ ↩² ↩³ ↩⁴ ↩⁵

22. Ye Wang, Jiahao Xun, Minjie Hong, Jieming Zhu, Tao Jin, Wang Lin, Haoyuan Li, Linjun Li, Yan Xia, Zhou Zhao, et al. 2024b. EAGER: Two-Stream Generative Recommender with Behavior-Semantic Collaboration. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 3245–3254. ↩ ↩²

23. Bowen Zheng, Yupeng Hou, Hongyu Lu, Yu Chen, Wayne Xin Zhao, Ming Chen, and Ji-Rong Wen. 2024. Adapting large language models by integrating collaborative semantics for recommendation. In 2024 IEEE 40th International Conference on Data Engineering (ICDE). IEEE, 1435–1448. ↩ ↩²

24. Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. 2022. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11523–11532. ↩

25. Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. 2022. Designing effective sparse expert models. arXiv preprint arXiv:2202.08906 2, 3 (2022), 17. ↩ ↩²

26. Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. 2022. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning. PMLR, 5547–5569. ↩ ↩²

27. Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. 2024. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066 (2024). ↩ ↩²

28. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems 36 (2024). ↩ ↩² ↩³ ↩⁴

29. Wentao Shi, Jiawei Chen, Fuli Feng, Jizhi Zhang, Junkang Wu, Chongming Gao, and Xiangnan He. 2023. On the theories behind hard negative sampling for recommendation. In Proceedings of the ACM Web Conference 2023. 812–822. ↩

30. Michael E Houle and Michael Nett. 2014. Rank-based similarity search: Reducing the dimensional dependence. IEEE transactions on pattern analysis and machine intelligence 37, 1 (2014), 136–150. ↩

31. Marius Muja and David G Lowe. 2014. Scalable nearest neighbor algorithms for high dimensional data. IEEE transactions on pattern analysis and machine intelligence 36, 11 (2014), 2227–2240. ↩

32. Anshumali Shrivastava and Ping Li. 2014. Asymmetric LSH (ALSH) for sublinear time maximum inner product search (MIPS). Advances in neural information processing systems 27 (2014). ↩

33. Tiezheng Ge, Kaiming He, Qifa Ke, and Jian Sun. 2013. Optimized product quantization. IEEE transactions on pattern analysis and machine intelligence 36, 4 (2013), 744–755. ↩

34. Herve Jegou, Matthijs Douze, and Cordelia Schmid. 2010. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence 33, 1 (2010), 117–128. ↩

35. Yubao Tang, Ruqing Zhang, Jiafeng Guo, and Maarten de Rijke. 2023. Recent advances in generative information retrieval. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region. 294–297. ↩

36. Yi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, et al. 2022. Transformer memory as a differentiable search index. Advances in Neural Information Processing Systems 35 (2022), 21831–21843. ↩ ↩² ↩³

37. Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2020. Autoregressive entity retrieval. arXiv preprint arXiv:2010.00904 (2020). ↩

38. Chao Feng, Wuchao Li, Defu Lian, Zheng Liu, and Enhong Chen. 2022. Recommender forest for efficient retrieval. Advances in Neural Information Processing Systems 35 (2022), 38912–38924. ↩ ↩²

39. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024). ↩

40. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems 33 (2020), 3008–3021. ↩

41. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35 (2022), 27730–27744. ↩

42. Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics. PMLR, 4447–4455. ↩

43. Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. 2024. Provably Robust DPO: Aligning Language Models with Noisy Feedback. In ICML 2024. ↩ ↩²

44. Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024b. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. arXiv preprint arXiv:2401.08417 (2024). ↩

45. Xinchen Luo, Jiangxia Cao, Tianyu Sun, Jinkai Yu, Rui Huang, Wei Yuan, Hezheng Lin, Yichen Zheng, Shiyao Wang, Qigen Hu, et al. 2024. QARM: Quantitative Alignment Multi-Modal Recommendation at Kuaishou. arXiv preprint arXiv:2411.11739 (2024). ↩ ↩²

46. Han Liu, Yinwei Wei, Xuemeng Song, Weili Guan, Yuan-Fang Li, and Liqiang Nie. 2024. MMGRec: Multimodal Generative Recommendation with Transformer Model. arXiv preprint arXiv:2404.16555 (2024). ↩

47. Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. 2021. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing 30 (2021), 495–507. ↩

48. Zhirui Kuai, Zuxu Chen, Huimu Wang, Mingming Li, Dadong Miao, Wang Binbin, Xusong Chen, Li Kuang, Yuxing Han, Jiaxing Wang, et al. 2024. Breaking the Hourglass Phenomenon of Residual Quantization: Enhancing the Upper Bound of Generative Retrieval. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track. 677–685. ↩

49. Shuyuan Xu, Wenyue Hua, and Yongfeng Zhang. 2024a. Openp5: An open-source platform for developing, training, and evaluating llm-based recommender systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 386–394. ↩