Not All the Same: Understanding and Informing Similarity Estimation in Tile-Based Video Games
Authors: 
Sebastian Berns, 
Vanessa Volz, 
Laurissa Tokarchuk, Sam Snodgrass, Christian GuckelsbergerAuthors Info & Claims
Sebastian Berns, Vanessa Volz, Laurissa Tokarchuk, Sam Snodgrass, Christian GuckelsbergerAuthors Info & ClaimsArticle No.: 366, Pages 1 - 23
Abstract
Similarity estimation is essential for many game AI applications, from the procedural generation of distinct assets to automated exploration with game-playing agents. While similarity metrics often substitute human evaluation, their alignment with our judgement is unclear. Consequently, the result of their application can fail human expectations, leading to e.g. unappreciated content or unbelievable agent behaviour. We alleviate this gap through a multi-factorial study of two tile-based games in two representations, where participants (N=456) judged the similarity of level triplets. Based on this data, we construct domain-specific perceptual spaces, encoding similarity-relevant attributes. We compare 12 metrics to these spaces and evaluate their approximation quality through several quantitative lenses. Moreover, we conduct a qualitative labelling study to identify the features underlying the human similarity judgement in this popular genre. Our findings inform the selection of existing metrics and highlight requirements for the design of new similarity metrics benefiting game development and research.
Supplemental Material
MP4 File - Video Presentation
Video Presentation
- Download
- 24.18 MB
- Transcript
CSV File - Study 1 Dataset
Study 1 survey (N=460) response data from Qualtrics, anonymised (CSV)
- Download
- 22.94 MB
ZIP File - Study Material
Study 1
- Download
- 13.44 MB
References
[1]
Sameer Agarwal, Josh Wills, Lawrence Cayton, Gert Lanckriet, David Kriegman, and Serge Belongie. 2007. Generalized Non-metric Multidimensional Scaling. In Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics(Proceedings of Machine Learning Research, Vol. 2). PMLR.
[2]
Alberto Alvarez, Steve Dahlskog, Jose Font, Johan Holmberg, and Simon Johansson. 2018. Assessing Aesthetic Criteria in the Evolutionary Dungeon Designer. In Proceedings of the 13th International Conference on the Foundations of Digital Games(FDG ’18). Association for Computing Machinery, New York, NY, USA.
[3]
Yoav Benjamini and Yosef Hochberg. 1995. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society: Series B (Methodological) 57, 1 (1995), 289–300.
[4]
Gerard R Bentley and Joseph C Osborn. 2019. The Videogame Affordances Corpus. In AIIDE Workshop on Experimental AI in Games (EXAG).
[5]
Debosmita Bhaumik, Julian Togelius, Georgios N Yannakakis, and Ahmed Khalifa. 2023. Lode Enhancer: Level Co-creation Through Scaling. In Proceedings of the 18th International Conference on the Foundations of Digital Games.
[6]
Alessandro Canossa and Gillian Smith. 2015. Towards a procedural evaluation technique: Metrics for level design. In The 10th International Conference on the Foundations of Digital Games. sn, 8.
[7]
Kate Compton and Michael Mateas. 2015. Casual Creators. In Proceedings of International Conference on Computational Creativity. 228–235.
[8]
Michael Cook, Jeremy Gow, Gillian Smith, and Simon Colton. 2021. Danesh: Interactive tools for understanding procedural content generators. IEEE Transactions on Games 14, 3 (2021), 329–338.
[9]
Çağatay Demiralp, Michael S. Bernstein, and Jeffrey Heer. 2014. Learning Perceptual Kernels for Visualization Design. IEEE Transactions on Visualization and Computer Graphics 20, 12 (2014).
[10]
Maria Edwards, Ming Jiang, and Julian Togelius. 2021. Search-Based Exploration and Diagnosis of TOAD-GAN. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Vol. 17. 140–147.
[11]
Gustav Theodor Fechner. 1860. Elemente der Psychophysik. Vol. 2. Breitkopf u. Härtel.
[12]
Matthew C. Fontaine, Ruilin Liu, Ahmed Khalifa, Jignesh Modi, Julian Togelius, Amy K. Hoover, and Stefanos Nikolaidis. 2021. Illuminating Mario Scenes in the Latent Space of a Generative Adversarial Network. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 5922–5930.
[13]
Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. 2023. DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data. In Advances in Neural Information Processing Systems, Vol. 36.
[14]
Jason Grinblat and Charles Brian Bucklew. 2010. Caves of Qud.
[15]
Christian Guckelsberger, Christoph Salge, Jeremy Gow, and Paul Cairns. 2017. Predicting player experience without the player. an exploratory study. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play. 305–315.
[16]
Manuel Lagunas, Sandra Malpica, Ana Serrano, Elena Garces, Diego Gutierrez, and Belen Masia. 2019. A Similarity Measure for Material Appearance. ACM Trans. Graph. 38, 4, Article 135 (2019), 12 pages.
[17]
Ming Li, Xin Chen, Xin Li, Bin Ma, and P.M.B. Vitanyi. 2004. The Similarity Metric. IEEE Transactions on Information Theory 50, 12 (2004).
[18]
Simon M. Lucas and Vanessa Volz. 2019. Tile Pattern KL-divergence for Analysing and Evolving Game Levels. In Proceedings of the Genetic and Evolutionary Computation Conference(GECCO ’19). Association for Computing Machinery, 170–178.
[19]
Emanuel Maiberg. 2016. ‘No Man’s Sky’ Is Like 18 Quintillion Bowls of Oatmeal. https://www.vice.com/en/article/nz7d8q/no-mans-sky-review. [Online; accessed 27-November-2023].
[20]
Julian Mariño, Willian Reis, and Levi Lelis. 2015. An empirical evaluation of evaluation metrics of procedurally generated Mario levels. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Vol. 11. 44–50.
[21]
Budiman Minasny and Alex B. McBratney. 2006. A Conditioned Latin Hypercube Method for Sampling in the Presence of Ancillary Information. Computers & Geosciences 32, 9 (2006).
[22]
Michal Piovarči, David IW Levin, Danny M Kaufman, and Piotr Didyk. 2018. Perception-aware modeling and fabrication of digital drawing tools. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1–15.
[23]
Michal Piovarči, David I. W. Levin, Jason Rebello, Desai Chen, Roman Ďurikovič, Hanspeter Pfister, Wojciech Matusik, and Piotr Didyk. 2016. An Interaction-Aware, Perceptual Model for Non-Linear Elastic Objects. ACM Trans. Graph. 35, 4, Article 55 (jul 2016).
[24]
Robert Gilmore Pontius and Marco Millones. 2011. Death to Kappa: Birth of Quantity Disagreement and Allocation Disagreement for Accuracy Assessment. International Journal of Remote Sensing 32, 15 (2011).
[25]
Younès Rabii and Michael Cook. 2023. Why Oatmeal is Cheap: Kolmogorov Complexity and Procedural Generation. In Proceedings of the 18th International Conference on the Foundations of Digital Games. 1–7.
[26]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning. PMLR.
[27]
Ruben Rodriguez Torrado, Ahmed Khalifa, Michael Cerny Green, Niels Justesen, Sebastian Risi, and Julian Togelius. 2020. Bootstrapping Conditional GANs for Video Game Level Generation. In 2020 IEEE Conference on Games (CoG). 41–48.
[28]
Bernice E Rogowitz, Thomas Frese, John R Smith, Charles A Bouman, and Edward B Kalin. 1998. Perceptual image similarity experiments. In Human Vision and Electronic Imaging III, Vol. 3299. SPIE, 576–590.
[29]
Anurag Sarkar and Seth Cooper. 2022. tile2tile: Learning Game Filters for Platformer Style Transfer. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Vol. 18. 53–60.
[30]
Noor Shaker, Miguel Nicolau, Georgios N. Yannakakis, Julian Togelius, and Michael O’Neill. 2012. Evolving Levels for Super Mario Bros Using Grammatical Evolution. In 2012 IEEE Conference on Computational Intelligence and Games (CIG).
[31]
Weiqi Shi, Zeyu Wang, Cyril Soler, and Holly Rushmeier. 2021. A Low-Dimensional Perceptual Space for Intuitive BRDF Editing. In EGSR 2021-Eurographics Symposium on Rendering-DL-only Track. 1–13.
[32]
Julius Sim and Chris C Wright. 2005. The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements. Physical Therapy 85, 3 (2005).
[33]
Gillian Smith and Jim Whitehead. 2010. Analyzing the Expressive Range of a Level Generator. In Proceedings of the 2010 Workshop on Procedural Content Generation in Games(PCGames ’10). Association for Computing Machinery, New York, NY, USA, Article 4.
[34]
Oskar Stålberg, Richard Meredith, and Martin Kvale. 2018. Bad North.(2018). Plausible Concept (2018).
[35]
Adam Summerville. 2018. Expanding expressive range: Evaluation methodologies for procedural content generation. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Vol. 14. 116–122.
[36]
Adam Summerville, Julian R. H. Mariño, Sam Snodgrass, Santiago Ontañón, and Levi H. S. Lelis. 2017. Understanding Mario: An Evaluation of Design Metrics for Platformers. In Proceedings of the 12th International Conference on the Foundations of Digital Games. Association for Computing Machinery.
[37]
Adam Summerville, Sam Snodgrass, Matthew Guzdial, Christoffer Holmgård, Amy K Hoover, Aaron Isaksen, Andy Nealen, and Julian Togelius. 2018. Procedural Content Generation via Machine Learning (PCGML). IEEE Transactions on Games 10, 3 (2018), 257–270.
[38]
Adam Summerville, Sam Snodgrass, Michael Mateas, and Santiago Ontañón. 2016. The VGLC: The Video Game Level Corpus. In Proceedings of the 7th Workshop on Procedural Content Generation. arXiv:1606.07487
[39]
Graham Todd, Sam Earle, Muhammad Umair Nasir, Michael Cerny Green, and Julian Togelius. 2023. Level Generation Through Large Language Models. In Proceedings of the 18th International Conference on the Foundations of Digital Games(FDG ’23). Association for Computing Machinery, New York, NY, USA, Article 70, 8 pages.
[40]
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing Data Using T-SNE. Journal of Machine Learning Research 9, 86 (2008), 2579–2605.
[41]
Laurens van der Maaten and Kilian Weinberger. 2012. Stochastic Triplet Embedding. In 2012 IEEE International Workshop on Machine Learning for Signal Processing.
[42]
Vanessa Volz. 2019. Uncertainty Handling in Surrogate Assisted Optimisation of Games. Ph. D. Dissertation. Technische Universität Dortmund.
[43]
Vanessa Volz, Niels Justesen, Sam Snodgrass, Sahar Asadi, Sami Purmonen, Christoffer Holmgård, Julian Togelius, and Sebastian Risi. 2020. Capturing Local and Global Patterns in Procedural Content Generation via Machine Learning. In 2020 IEEE Conference on Games (CoG).
[44]
Jan B. Vornhagen, April Tyack, and Elisa D. Mekler. 2020. Statistical Significance Testing at CHI PLAY: Challenges and Opportunities for More Transparency. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play(CHI PLAY ’20). 4–18.
[45]
Josh Wills, Sameer Agarwal, David Kriegman, and Serge Belongie. 2009. Toward a perceptual space for gloss. ACM Transactions on graphics (TOG) 28, 4 (2009), 1–15.
[46]
Oliver Withington and Laurissa Tokarchuk. 2023. The Right Variety: Improving Expressive Range Analysis with Metric Selection Methods. In Proceedings of the 18th International Conference on the Foundations of Digital Games(FDG ’23). Association for Computing Machinery, New York, NY, USA, Article 18, 11 pages.
[47]
Bang Wong. 2011. Points of View: Color Blindness. Nature Methods 8, 6 (2011).
[48]
Georgios N Yannakakis and Julian Togelius. 2011. Experience-driven procedural content generation. IEEE Transactions on Affective Computing 2, 3 (2011), 147–161.
[49]
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 586–595.
Index Terms
- Not All the Same: Understanding and Informing Similarity Estimation in Tile-Based Video Games
Recommendations
Towards Understanding Player Behavior in Blockchain Games: A Case Study of Aavegotchi
FDG '22: Proceedings of the 17th International Conference on the Foundations of Digital GamesBlockchain games introduce unique gameplay and incentive mechanisms by allowing players to be rewarded with in-game assets or tokens through financial activities. However, most blockchain games are not comparable to traditional games in terms of ...
The interplay between immersion and appeal in video games
A 2í 2 study on immersion and appeal of video games is presented.Immersion and appeal are found to be highly related.Immersion is found to be unaffected by playing experience.Appeal is found to be affected both by game played and playing experience. ...
Comments
Information & Contributors
Information
Published In
May 2024
18961 pages
ISBN:9798400703300
DOI:10.1145/3613904
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 May 2024
Check for updates
Badges
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Data Availability
Study Material: Study 1 https://dl.acm.org/doi/10.1145/3613904.3642077#pn3620-supplemental-material-1.zip
Study Material: Study 1 https://dl.acm.org/doi/10.1145/3613904.3642077#pn3620-supplemental-material-1.zip
Study Material: Study 1 https://dl.acm.org/doi/10.1145/3613904.3642077#pn3620-supplemental-material-1.zip
Funding Sources
- EPSRC Centre for Doctoral Training in Intelligent Games and Game Intelligence
Conference
Acceptance Rates
Overall Acceptance Rate 6,199 of 26,314 submissions, 24%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 221Total Downloads
- Downloads (Last 12 months)221
- Downloads (Last 6 weeks)41
Reflects downloads up to 12 Sep 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderFull Text
View this article in Full Text.
Full TextHTML Format
View this article in HTML Format.
HTML Format