2nd International Workshop on Industrial Recommendation Systems

August 15 11:00pm (SG) - August 16 8:00am (SG), 2021 / August 15 8:00am (PT) - August 15 5:00pm (PT), 2021


Recommendation systems are used widely in ecommerce industries, multimedia content platforms and social networks to provide suggestions that a user will most likely consume or connect; thus, improving the user experience. This motivates people in both industry and research organizations to focus on personalization or recommendation algorithms, which has resulted in a plethora of research papers. While academic research mostly focuses on the performance of recommendation algorithms in terms of ranking quality or accuracy, it often neglects key factors that impact how a recommendation system will perform in a real-world environment. These key factors include but are not limited to: business metric definition and evaluation, recommendation quality control, data and model scalability, model interpretability, model robustness and fairness, and resource limitations, such as budget on computing and memory resources, engineering workforce cost, etc..

The gap in constraints and requirements between academic research and industry limits the broad applicability of many of academia’s contributions for industrial recommendation systems. This workshop aspires to bridge this gap by bringing together researchers from both academia and industry. Its goal is to serve as a platform via which academic researchers become aware of the additional factors that may affect the chances of algorithm adoption into real production systems, and the performance of the algorithms if deployed. Industrial researchers will also benefit from sharing the practical frameworks at an industrial level.

Registration Information:

Please note that all workshop attendees must be registered, either into the whole conference, or in a workshop only registration. Please refer to the KDD2021 main conference for more information regarding the registration.

Regarding to COVID-19:

Following the schedule of the KDD2021 conference, our workshop will be fully virtual. More details will be shared as soon as we are able to.

Camera Ready Version Submission Instructions (Due Date: July 18, 2021):

Please submit your camera ready version using your easychair submission by updating the pdf file with your camera ready pdf. Please contact the co-chairs if you are not able to update the pdf file.

Oral Presentation and Poster Preparation Instructions:

This year, we have 11 papers accepted out of 23 submissions, within which 5 papers are selected for Oral Presentation. All papers accepted are required to have a poster presentation.
  1. Accepted oral papers: please be prepared for a 15 minutes oral session, with a 10 - 12 minutes presentation and 3 - 5 minutes QA. The 10 - 12 minutes presentation needs to be pre-recorded as a video as required by KDD2021.
  2. Accepted poster papers: please prepare a 15-minutes video. This video will be posted on our website before the workshop date.
  3. Please record your video in .mp4 format and submit it before August 1, 2021. Please check your email for submission instructions.


Xiangnan He

Xiangnan He Professor USTC

Christoph Kofler

Christoph Kofler Senior Manager of Product Innovation Netflix

Rishabh Mehrotra

Rishabh Mehrotra Staff Research Scientist Spotify

Yunjiang Jiang Principal Research Scientist JD.com

Da Xu

Da Xu Senior Machine Learning Engineer WalmartLabs

Anoop Deoras

Anoop Deoras Group Manager Amazon AWS

Even Oldridge

Even Oldridge Senior Manager - Recsys Platform Nvidia

Important Dates

May 20, 2021 May 30, 2021 : Workshop paper submission (deadline extended)

June 20, 2021 June 25, 2021 : Workshop paper notifications

July 18, 2021: Camera-ready deadline for workshop papers

August 15 11:00pm (SG) - August 16 8:00am (SG), 2021 / August 15 8:00am (PT) - August 15 5:00pm (PT), 2021: Workshop Date


Time Speaker Title
10:55PM - 11:00PM, 2021/08/15 (SG)
07:55AM - 08:00AM, 2021/08/15 (PT)
Host Chair Welcome and Open Remarks
11:00PM - 11:40PM, 2021/08/15 (SG)
08:00AM - 08:40AM, 2021/08/15 (PT)
Xiangnan He [USTC] Keynote 1: Addressing Data Bias and Feedback Loop
in Recommemdation with Causal Inference [YouTube]
11:40PM - 00:20AM, 2021/08/15-16 (SG)
08:40AM - 09:20AM, 2021/08/15 (PT)
Christoph Kofler [Netflix] Keynote 2: Challenges in Industrial Recommendation Systems [YouTube]
00:20AM - 01:00AM, 2021/08/16 (SG)
09:20AM - 10:00AM, 2021/08/15 (PT)
Rishabh Mehrotra [Spotify] Keynote 3 [Live Presentation]
01:00AM - 01:10AM, 2021/08/16 (SG)
10:00AM - 10:10AM, 2021/08/15 (PT)
N/A Break
01:10AM - 01:25AM, 2021/08/16 (SG)
10:10AM - 10:25AM, 2021/08/15 (PT)
Cheng Jie, et al. Oral 1: Bidding via Clustering Ads Intentions:
an Efficient Search Engine Marketing System for E-commerce [YouTube]
01:25AM - 01:40AM, 2021/08/16 (SG)
10:25AM - 10:40AM, 2021/08/15 (PT)
Nagaraj Kota, et al. Oral 2: Learnings from Building the User Intent Embedding Store
towards Job Personalization at LinkedIn [YouTube]
01:40AM - 02:20AM, 2021/08/16 (SG)
10:40AM - 11:20AM, 2021/08/15 (PT)
Panelists: Keynote Speakers Panel Discussion [Live Presentation]
02:20AM - 03:00AM, 2021/08/16 (SG)
11:20AM - 12:00PM, 2021/08/15 (PT)
Yunjiang Jiang [JD.com] Keynote 4: A deep dive through JD Search [YouTube]
03:00AM - 04:00AM, 2021/08/16 (SG)
12:00PM - 01:00PM, 2021/08/15 (PT)
N/A Lunch Break
04:00AM - 04:40AM, 2021/08/16 (SG)
01:00PM - 01:40PM, 2021/08/15 (PT)
Da Xu [Walmart] Keynote 5: A Deep Dive of Embedding for E-commerce ML [YouTube]
04:40AM - 05:20AM, 2021/08/16 (SG)
01:40PM - 02:20PM, 2021/08/15 (PT)
Anoop Deoras [Amazon AWS] Keynote 6: Personalization For The World [YouTube]
05:20AM - 06:00AM, 2021/08/16 (SG)
02:20PM - 03:00PM, 2021/08/15 (PT)
Even Oldridge [Nvidia] Keynote 7: Moving Beyond Recommender Models [YouTube]
06:00AM - 06:10AM, 2021/08/16 (SG)
03:00PM - 03:10PM, 2021/08/15 (PT)
N/A Break
06:10AM - 06:25AM, 2021/08/16 (SG)
03:10AM - 03:25AM, 2021/08/15 (PT)
Nicolò Felicioni, et al. Oral 3: Measuring the Ranking Quality of Recommendations
in a Two-Dimensional Carousel Setting [YouTube]
06:25AM - 06:40AM, 2021/08/16 (SG)
03:25PM - 03:40PM, 2021/08/15 (PT)
Haochen Liu, et al. Oral 4: Self-supervised Learning for Alleviating Selection Bias
in Recommendation Systems [YouTube]
06:40AM - 06:55AM, 2021/08/16 (SG)
03:40PM - 03:55PM, 2021/08/15 (PT)
Wei Xiao, et al. Oral 5: Two-stage Voice Application Recommender System
for Unhandled Utterances in Intelligent Personal Assistant [YouTube]
06:55AM - 07:00AM, 2021/08/16 (SG)
03:55PM - 04:00PM, 2021/08/15 (PT)
Justin Basilico [Netflix] Closing Remarks
07:00AM - 08:00AM, 2021/08/16 (SG)
04:00PM - 05:00PM, 2021/08/15 (PT)
Authors of Accepted Papers Poster Session: for details please attend the workshop
[Live Presentation]

Accepted Publications

  • Bidding via Clustering Ads Intentions: an Efficient Search Engine Marketing System for E-commerce [PDF] [YouTube] [from Walmart] (Oral)
  •      Authors: Cheng Jie, Da Xu, Zigeng Wang, Lu Wang and Wei Shen

  • Embedding-based Recommender System for Job to CandidateMatching on Scale [PDF] [YouTube] [from CareerBuilder]
  •      Authors: Jing Zhao, Jingya Wang, Madhav Sigdel, Bopeng Zhang, Phuong Hoang, Mengshu Liu and Mohammed Korayem

  • Learnings from Building the User Intent Embedding Store towards Job Personalization at LinkedIn [PDF] [YouTube] [from LinkedIn AI, India] (Oral)
  •      Authors: Nagaraj Kota, Venkatesh Duppada, Ashvini Jindal and Mohit Wadhwa

  • Measuring the Ranking Quality of Recommendations in a Two-Dimensional Carousel Setting [PDF] [YouTube] [from ContentWise and Polytechnic University of Milan] (Oral)
  •      Authors: Nicolò Felicioni, Maurizio Ferrari Dacrema, Fernando Benjamín Pérez Maurera and Paolo Cremonesi

  • Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation [PDF] [YouTube] [from Hanjuku-kaso, ZOZO Technologies and Yale University]
  •      Authors: Yuta Saito, Shunsuke Aihara, Megumi Matsutani and Yusuke Narita

  • Personalized Compatibility Metric Learning [PDF] [YouTube] [from Amazon and UCLA]
  •      Authors: Meet Taraviya, Anurag Beniwal, Yen-Liang Lin and Larry Davis

  • Personalized Embedding-based e-Commerce Recommendations at eBay [PDF] [YouTube] [from eBay]
  •      Authors: Tian Wang, Yuri M. Brovman and Sriganesh Madhvanath

  • Self-supervised Learning for Alleviating Selection Bias in Recommendation Systems [PDF] [YouTube] [from ByteDance and Michigan State University] (Oral)
  •      Authors: Haochen Liu, Da Tang, Ji Yang, Xiangyu Zhao, Jiliang Tang and Youlong Cheng

  • Semi-Explicit MMoE via Heterogeneous Multi-Task Learning for Ranking Relevance [PDF] [YouTube] [from JD.com and Tianjin University]
  •      Authors: Ziyang Liu, Junqing Chen, Yunjiang Jiang, Yue Shang, Wei Xiong, Sulong Xu, Zhaomeng Cheng, Bo Long, Lingfei Wu, Yun Xiao and Di Jin

  • Two-stage Voice Application Recommender System for Unhandled Utterances in Intelligent Personal Assistant [PDF] [YouTube] [from AWS AI and Alexa AI] (Oral)
  •      Authors: Wei Xiao, Qian Hu, Thahir Mohamed, Zheng Gao, Xibin Gao, Radhika Arava and Mohamed Abdelhady

  • With One Voice: Composing a Travel Voice Assistant from Repurposed Models [PDF] [YouTube] [from Booking.com]
  •      Authors: Shachaf Poran, Gil Amsalem, Amit Beka and Dmitri Goldenberg

    Program Committee

    We want to thank our program committee members for their hard work to select the valuable publications for our workshop.

    • Arushi Prakash, Zulily
    • Athanasios N. Nikolakopoulos, Amazon
    • Da Xu, WalmartLabs
    • Emma Kong, Netflix
    • Erik Schmidt, Netflix
    • Flavien Prost, Google
    • Ivan Ji, TikTok
    • Jing Lu, WalmartLabs
    • Khoa Doan, University of Maryland
    • Luyi Ma, WalmartLabs
    • Mansi Mane, Amazon
    • Mengting Wan, Microsoft
    • Pradeep Ranganathan, University Of Michigan
    • Qi Shen, Tongji University
    • Rein Houthooft, Netflix Research
    • Sinduja Subramaniam, WalmartLabs
    • Sudarshan Lamkhede, Netflix Research
    • Sushant Kumar, Walmartlabs
    • Venugopal Mani, WalmartLabs
    • Xiaohan Li, University of Illinois at Chicago
    • Yao ,Zhou University of Illinois at Urbana-Champaign
    • Yiming Zhang, Tongji University
    • Yitong Pang, Tongji University
    • Yuanqi Du, George Mason University
    • Zezhong Zhang, EBay
    • Zhiwei Wang, University of Illinois at Chicago

    Workshop Co-Chairs

    Jianpeng Xu

    Jianpeng Xu WalmartLabs

    Lingfei Wu

    Lingfei Wu JD.com

    Linsey Pang

    Linsey Pang WalmartLabs

    Mohit Sharma

    Mohit Sharma Google

    Organizing Committees

    Dawei Yin

    Dawei Yin Baidu

    George Karypis

    George Karypis University of Minnesota Twin Cities

    Justin Basilico

    Justin Basilico NetFlix

    Philip S. Yu

    Philip S. Yu University of Illinois at Chicago

    Call For Papers

    This workshop welcomes submissions from researchers and industrial practitioners broadly related to recommendation systems, such as novel recommendation models, efficient recommendation algorithms, novel industrial frameworks, etc. In order to emphasize the gap between the two communities, we extremely welcome submissions on industrial recommendation system infrastructures based on given resources, models and algorithms supported by the specific infrastructures, and frameworks or end-to-end systems that have been deployed in real world production. All accepted papers will be posted on the workshop website and will not appear in the KDD proceedings. Selected papers will be presented as contributed talks.

    Specific topics of interest are including but not limited to:

    1. Frameworks or end-to-end systems from industry are extremely welcomed.
    2. Novel data mining and machine learning algorithms for scalable Recommender systems.
    3. Personalization, including personalized product recommendation, streaming content recommendation, ads recommendation, news and article recommendation, etc.
    4. New applications related to recommendation systems.
    5. Existing or novel infrastructures for recommendation systems.
    6. Interactive recommendation system.
    7. Explainability of recommendations.
    8. Fairness, privacy and security in recommender systems.
    9. Recommendations under multi-objective and constraints.
    10. Reproducibility of models and evaluation metrics.
    11. Unbiased recommendation.
    12. User research studies on real-world recommender systems.
    13. Business impact of recommendation systems.

    Submission Directions:

    The workshop accepts long papers (limited to 9 pages), short papers (6 pages), posters (4 pages), abstracts and demos (2 pages). Paper submission and reviewing will be following the directions of the KDD main conference. Reviews are not double-blind, and author names and affiliations should be listed. Submissions should include all content and references within the limited pages, and must be in PDF format and formatted according to the new Standard ACM Conference Proceedings Template. For LaTeX users: unzip acmart.zip, make, and use sample-sigconf.tex as a template. Additional information about formatting and style files is available online at: https://www.acm.org/publications/proceedings-template. Papers that do not meet the formatting requirements will be rejected without review. In addition, authors can provide an optional two (2) page supplement at the end of their submitted paper (it needs to be in the same PDF file) focused on reproducibility.

    Please submit your paper through this Easychair link. Please reach out to irs-kdd@googlegroups.com or irs2021-0@easychair.org for any questions.