1st International Workshop on Industrial Recommendation Systems


Recommendation systems are used widely in eCommerce industries and multimedia content platforms to provide suggestions that a user will most likely consume; thus, improving the user experience. This motivates people in both industry and research organizations to focus on personalization or recommendation algorithms, which has resulted in a plethora of research papers. While academic research mostly focuses on the performance of recommendation algorithms in terms of ranking quality or accuracy, it often neglects key factors that impact how a recommendation system will perform in a real-world environment. These key factors include but are not limited to: data and model scalability, model serving latency, model interpretability, and resource limitations, such as budget on compute and memory resources, engineering workforce cost, etc.

The gap in constraints and requirements between academic research and industry limits the broad applicability of many of academia’s contributions for industrial recommendation systems. This workshop aspires to bridge this gap by bringing together researchers from both academia and industry. Its goal is to serve as a platform via which academic researchers become aware of the additional factors that may affect the chances of algorithm adoption into real production systems, and the performance of the algorithms if deployed. Industrial researchers will also benefit from sharing the practical frameworks at an industrial level.

Attending the workshop:

As registered attendees, you must have received the email from KDD conference regarding the log-in information to the virtual conference. Here is the link which will lead you directly to the workshop: IRS2020 and here is the link to the chat room of IRS2020: IRS2020 Chat Room . You can also search for the workshop using the keyword "irs2020" in the search box on top of the virtual conference page. We will announce the main zoom link to the workshop in the chat room on or before August 24, 2020. Invited keynotes and oral presentations will be given in the main zoom link, and the poster sessions will be hosted seprately with one zoom link for each poster. The zoom links will be shared within the chat room as well. We will also update the website to share the zoom links before the workshop. It is recommended that speakers would join the zoom room at their allocated time slot and attend the QA after their presentation.

In order to reduce the risk of network interruptions, pre-recorded videos of the presentations will be played via zoom, if the video is available. Otherwise, the speaker will deliver the talk live. For poster sessions, the authors will determine their format of presentation of their work in the provided zoom link (slides/poster/short video, etc). In general, it is recommended that people watch the pre-recorded video first and join the poster session for Q&A.

Registration Information:

Please note that all workshop attendees must be registered, either into the whole conference, or in a workshop only registration. The deadline for registration is August 20, 2020. If you have been registered into the full KDD2020 conference, please ignore the rest of the message. If you are planning to register only into workshops and tutorials, please use this link for registration.

Regarding to COVID-19:

Following the schedule of the KDD2020 conference, our workshop will be fully virtual. More details will be shared as soon as we are able to.

Camera Ready Version Submission Instructions (Due Date: July 5, 2020):

Please upload your camera ready version of your submission using this Google Form.

Oral Presentation and Poster Preparation Instructions:

  1. Accepted oral papers: please be prepared for a 20-minutes presentation, with a 15-minutes video and 5-minutes QA.
  2. Accepted poster papers: please prepare a 15-minutes video. This video will be posted on our website before the workshop date.
  3. All videos are due on July 31, 2020.
  4. Please read this documentation for video preparation instructions.
  5. The videos for IRS2020 should be uploaded to IRSKDD'20 folder under KDD2020 Workshops folder. See the above documentation for details.
  6. We will share the virtual conference room links for each poster and oral paper for QA sessions prior to workshop.

PROGRAM (August 24, 2020)

Zoom link to the IRS 2020: Please check the chat room of IRS2020 for the link

Q&A: Please use Zoom group chat or Whova chat to type/send your questions for a talk. Speaker will answer them at the end of the talk. Please mention speaker or title if the question is not for the live session.

Link to the virtual conference page: IRS2020

Link to the chat room of IRS2020: IRS2020 Chat Room

Link to Whova page for IRS2020: Whova IRS2020

Time Speaker Title
8:00AM - 8:45AM Dr. Liangjie Hong [LinkedIn] Keynote - Recent Challenges and Advances in Industrial Recommender Systems [YouTube]
8:45AM - 9:30AM Dr. Ed Chi [Google] Keynote - Beyond Being Accurate: Solving Real-World Recommendation Problems with Neural Models
9:30AM - 10:00AM N/A Poster Session / Coffee Break [YouTube]
10:00AM - 10:20AM Authors from Twitter Oral paper - Lessons Learned Addressing Dataset Bias in Model-Based Candidate Generation at Twitter [YouTube]
10:20AM - 10:40AM Authors from Expedia Oral paper - Landing Page Personalization at Expedia Group [YouTube]
10:40AM - 11:00AM Authors from Nvidia Oral paper - Merlin: A GPU Accelerated Recommendation Framework [YouTube]
11:00AM - 11:15AM N/A Break / Social / QA
11:15AM - 12:00PM Dr. George Karypis [UMN/Amazon] Keynote - Graph Neural Networks, DGL, and Applications


George Karypis George Karypis is a Professor at the Department of Computer Science & Engineering at the University of Minnesota in the Twin Cities of Minneapolis and Saint Paul, and a Senior Principal Scientist at Amazon AWS. He is renowned for seminal contributions in the areas of data mining, recommender systems, and high-performance computing. He has brought his innovative research ideas into practice via a wide-range of high-quality software packages, such as CLUTO and the METIS family of serial and parallel partitioning systems. His software has been incorporated into well over 200 different commercial software systems used by millions of people worldwide and in several hundred academic and government codes. Some notable companies using his work include Amazon, Netflix, Spotify, and Comcast. He has also authored and co-authored a large number of highly cited papers in these areas on topics related to clustering, graph mining, pattern discovery, collaborative filtering, and graph partitioning. Google Scholar shows over 72,000 citations to his work and his h-index is 94. He has received numerous awards including the "IEEE ICDM 10-Year Highest-Impact Paper Award" for his work that developed computationally efficient algorithms to mine large graph databases. He also recently resceived IEEE ICDM Research Contributions Award, which is IEEE’s highest recognition for research achievements in Data Mining. Furthermore, he received the International World Wide Web Conference's "Seoul Test of Time Award" for his work that pioneered an entirely new way of building recommender systems that exploit relations between items.

Liangjie Hong Liangjie Hong is the Director of Engineering, AI at LinkedIn Corporation. He manages multiple teams of machine learning engineers and applied researchers, driving AI solutions to a wide range of LinkedIn core businesses such as Job Search, Job-You-May-Be-Interested-In (JYMBII), Guest Search, Recruiter Search, Recommended Matches and Job Notifications. As an industrial researcher, he have published papers in all major international conferences in data mining and applied machine learning including SIGIR, WWW, KDD, CIKM, AAAI, WSDM, RecSys and ICML with more than 3,500 citations (H-index: 19), winning WWW 2011 Best Poster Paper Award, WSDM 2013 Best Paper Nominated and RecSys 2014 Best Paper Award.

Ed H. Chi Ed H. Chi is a Principal Scientist at Google, leading several machine learning research teams focusing on neural modeling, inclusive ML, reinforcement learning, and recommendation systems in Google Brain . He has delivered significant improvements for YouTube, News, Ads, Google Play Store at Google with >170 product launches in the last 3 years. With 39 patents and over 120 research articles, he is also known for research on user behavior in web and social media.

Accepted Posters (selected posters for Oral Presentation)

  • A Flexible Large-Scale Similar Product Identification System in E-commerce [PDF] [YouTube]
  •      Authors: Zhen Zuo, Lixi Wang, Michinari Momma, Wenbo Wang, Yikai Ni, Jianfeng Lin and Yi Sun

  • Automated Mechanism to Choose Between Reinforcement Learning and Contextual Bandit in Website Personalization [PDF] [YouTube]
  •      Authors: Abhimanyu Mitra, Afroza Ali, Xiaotong Suo, Kailing Wang and Kannan Achan

  • Diversification of Complementary Item Recommendations with User Preference in Online Grocery [PDF][YouTube]
  •      Authors: Luyi Ma, Nimesh Sinha, Jason H.D. Cho, Sushant Kumar and Kannan Achan

  • Landing Page Personalization at Expedia Group [PDF][YouTube] (Oral)
  •      Authors: Pavlos Mitsoulis-Ntompos, Dionysios Varelas, Travis Brady, J. Eric Landry, Robert F. Dickerson, Timothy Renner, Chris Harris, Shuqin Ye, Abbas Amirabadi, Lisa Jones and Javier Luis Cardo

  • Lessons Learned Addressing Dataset Bias in Model-Based Candidate Generation at Twitter [PDF][YouTube] (Oral)
  •      Authors: Alim Virani, Jay Baxter, Dan Shiebler, Philip Gautier, Shivam Verma, Apoorv Sharma, Yan Xia, Chenguang Yu, Sumit Binnani and Linlin Chen

  • MBCAL: Sample Efficient and Variance Reduced Reinforcement Learning for Recommender Systems [PDF][YouTube]
  •      Authors: Fan Wang, Xiaomin Fang, Lihang Liu, Hao Tian and Zhiming Peng

  • Merlin: A GPU Accelerated Recommendation Framework [PDF] [YouTube](Oral)
  •      Authors: Even Oldridge, Julio Perez, Ben Frederickson, Minseok Lee, Zehuan Wang, Lei Wu, Fan Yu, Rick Zamora, Onur Yilmaz, Alec Gunny, Nicolas Koumchatzky and Vinh Nguyen

  • Multi-sided Exposure Bias in Recommendation [PDF][YouTube]
  •      Authors: Himan Abdollahpouri and Masoud Mansoury

  • Personalizing Multi-Modal Content for a Diverse Audience: A Scalable Deep Learning Approach [PDF][YouTube]
  •      Authors: Nishant Oli, Aditya Patel, Vishesh Sharma, Sai Dinesh Dacharaju and Sushrut Ikhar

  • Scalable and Personalized Item Recommendations Framework [PDF][YouTube]
  •      Authors: Nimesh Sinha, Selene Xu, Swati Bhatt, Abhinav Mathur, Jason H.D. Cho, Sushant Kumar and Kannan Achan

    Important Dates

    May 20 May 31, 2020 : Workshop paper submission (deadline extened)

    June 15, 2020: Workshop paper notifications

    July 05, 2020: Camera-ready deadline for workshop papers

    August 24, 2020 (8:00 AM - 12:00 PM): Workshop Date

    Call For Papers

    This workshop welcomes submissions from researchers and industrial practitioners broadly related to recommendation systems, such as novel recommendation models, efficient recommendation algorithms, novel industrial frameworks, etc. In order to emphasize the gap between the two communities, we extremely welcome submissions on industrial recommendation system infrastructures based on given resources, models and algorithms supported by the specific infrastructures, and frameworks or end-to-end systems that have been deployed in real world production. All accepted papers will be posted on the workshop website and will not appear in the KDD proceedings. Selected papers will be presented as contributed talks.

    Specific topics of interest are including but not limited to:

    1. Frameworks or end-to-end systems from industry are extremely welcomed.
    2. Novel data mining and machine learning algorithms for scalable Recommender systems.
    3. Personalization, including personalized product recommendation, streaming content recommendation, ads recommendation, news and article recommendation, etc.
    4. New applications related to recommendation systems.
    5. Existing or novel infrastructures for recommendation systems.
    6. Approaches to handling practical challenges like feedback loops between recommendation algorithm outputs and how people respond to them.
    7. Explainability of recommendations.
    8. Fairness in recommender systems.
    9. Recommendations under multi-objective and constraints.
    10. Reproducibility of models and evaluation metrics.
    11. Unbiased Recommendation

    Submission Directions:

    The workshop accepts long papers (limited to 9 pages), short papers (6 pages), posters (4 pages), abstracts and demos (2 pages). Paper submission and reviewing will be following the directions of the KDD main conference. Reviews are not double-blind, and author names and affiliations should be listed. Submissions should include all content and references within the limited pages, and must be in PDF format and formatted according to the new Standard ACM Conference Proceedings Template. For LaTeX users: unzip acmart.zip, make, and use sample-sigconf.tex as a template. Additional information about formatting and style files is available online at: https://www.acm.org/publications/proceedings-template. Papers that do not meet the formatting requirements will be rejected without review. In addition, authors can provide an optional two (2) page supplement at the end of their submitted paper (it needs to be in the same PDF file) focused on reproducibility.

    Please submit your paper through this Easychair link. Please reach out to irs2020-0@easychair.org for any questions.

    Program Committee

    We want to thank our program committee members for their hard work to select the valuable publications for our workshop.

    • Ajinkya More, Netflix
    • Athanasios Nikolakopoulos, University of Minnesota
    • Arushi Prakash, Zulily
    • Da Xu, WalmartLabs
    • Erik Schmidt, Netflix
    • Flavien Prost, Google
    • Gary Tang, Netflix
    • Guruprasad Nayak, Amazon
    • Khoa Doan, Virginia Tech
    • Konstantina Christakopoulou, Google
    • Mansi Mane, WalmartLabs
    • Pradeep Ranganathan, Lyft
    • Saurav Manchanda, University of Minnesota
    • Suhas Ranganath, Walmart Labs
    • Swayambhoo Jain, InterDigital Inc.
    • Xiaohan Li, UIC
    • Yao Zhou, UIUC
    • Yves Raimond, Netflix
    • Zhiwei Liu, UIC


    Jianpeng Xu

    Jianpeng Xu WalmartLabs

    Mohit Sharma

    Mohit Sharma Google

    Justin Basilico

    Justin Basilico NetFlix

    Dawei Yin

    Dawei Yin Baidu

    George Karypis

    George Karypis University of Minnesota Twin Cities

    Philip S. Yu

    Philip S. Yu University of Illinois at Chicago

    Please reach out to irs2020-0@easychair.org for any questions.