Time: 10:30-11:45, Mondays and Wednesdays Location: Online, using Zoom - the meeting link will be distributed later.
Shin Yoo email@example.com Office: E3-1 Room 2405
All class announcements, as well as Q&A, will take place using a dedicated Slack workspace. You are required to join cs453-2021-kaist.slack.com if you want to continue this course. It is strongly recommended that you install either a desktop or a mobile client, to get notifications. Email the lecturer or one of the TAs to get the invitation link, if you do not have one. When you sign up, please set your username as your real full name in English, followed by “(your student id number)”. For example, “Shin Yoo (20201234)”.”
This course is concerned with a broad range of software testing techniques, with a heavy emphasis on automation, tools, and frameworks, as well as the research outputs behind them. The topic will include, but are not limited to: black box testing/combinatorial testing, random testing, concepts of coverage, structural testing, mutation testing, regression testing, testability transformation, automated debugging, etc.
- Strong programming skills: you are required to actively contribute to group and individual project, which involves serious implementation. There will be also a number of hands-on sessions where we will program together during the class.
- Unix/Linux-savvy: you should be familiar with the usual build tools and Unix/Linux command line environments.
- Git-aware: knowing how to use git is mandatory for this course. First, we will use GitHub classroom for coursework. Second, you will be required to submit a github repository as part of your project deliverable.
- Ideally, CS350 Introduction to Software Engineering.
Please note that, unlike previous years, we will have the final exam instead of the mid-term exam. Also, there is no participation points.
- Coursework: 40%
- Final Exam: 30%
- Project: 30%
- Sungmin Kang (firstname.lastname@example.org)
We do not have a textbook per se, and the course will be based on slides and other reading material that are deemed appropriate. However, if you want to get broader sense for some of the topics dealt by this course, I recommend the following books and publications.
- Paul Ammann and Jeff Offutt. Introduction to Software Testing (2nd Ed.)
- Andreas Zeller. Why Programs Fail (2nd Ed.)
- Y. Jia and M. Harman. An analysis and survey of the development of mutation testing. IEEE transactions on software engineering, 37(5):649–678.
- P. McMinn. Search-based software test data generation: A survey. Software Testing, Verification and Reliability, 14(2):105–156, June 2004.
The schedules are tentative at the moment.
- 01 Mar: No Lecture (Public Holiday)
- 03 Mar: Introduction
- 08 Mar: Testing Fundamentals
- 10 Mar: Tutorial - Metaprogramming 101 for Python
- 15 Mar: Black Box Testing & Combinatorial Interaction Testing
- 17 Mar: Testing Finite State Machines
- 22 Mar: Control and Data Flow
- 24 Mar: Random and Adaptive Random Testing
- 29 Mar: Random Testing Hands-on
- 31 Mar: Mutation Testing
- 05 Apr: Mutation Testing Hands-on with PIT
- PIT: a practical mutation testing tool for Java
- 07 Apr: Regression Testing
- 12 Apr: Group Project Proposal
- 14 Apr: Group Project Proposal
- 19 Apr: No Lecture - Midterm Week
- 21 Apr: No Lecture - Midterm Week
- 26 Apr: Fault Localisation
- 28 Apr: Literature Review & Discussion - Fault Localisation
- 03 May: Search Based Test Data Generation
- 05 May: No Lecture (Children’s Day)
- 10 May: Non-testable Programs and Metamorphic Testing
- 12 May: Project Milestone Day
- 17 May: Web Testing Automation Hands-on
- 19 May: No Lecture (Happy Birthday, Buddha!)
- 24 May: Test Flakiness
- 26 May: No Lecture (ICSE 2021 Week)
- 31 May: Code Defenders - Mutation Testing Game
- 02 Jun: Testing Deep Neural Networks
- 07 Jun: Project Presentation
- 09 Jun: Project Presentation
- 14 Jun: Final Exam Week
- 16 Jun: CS453 Final Exam
Assignment 0: GitHub Classroom Onboarding
You need to get familiar with GitHub Classroom: create a GitHub account if you do not have one, and learn the basics of Git. The assignment invitation link is here.
Assignment 1: Introduction to Metaprogramming
You will learn how to manipulate Python code using
ast module. This assignment takes up 5% of total course grade. The assignment invitation link is here.
Assignment 2: Python Coverage Profiler
Your task is to write a coverage profiler for Python that can measure statement, branch, and condition coverage. This assignment takes up 15% of total course grade. The assignment invitation link is here.
Assignment 3: Mutation Testing
Your task is to write a full mutation testing tool that mutates the give Python code, executes the given test cases against the generated mutants, and finally produces kill matrices. This assignment takes up 10% of total course grade. The assignment link is here
Assignment 4: Delta Debugging
Your task will be to implement a delta debugging tool that minimises an error-revealing input. First, we will implement a linear and recursive DD for fake input. Subsequently, we will move onto Hierarchical Delta Debutting for Python programs (i.e., working with ASTs). The assignment link is here.
All teams should develop and/or implement an automated software testing technique based on an idea discussed during the course. I would encourage teams to pursue a novel idea, but a faithful reproduction of a state-of-the-art technique with solid evaluation would also do. If you are uncertain about your team’s idea, I will be happy to discuss it.
All teams will give a presentation on 20th and 22nd April to explain their project topics. I expect three things to be described clearly in the talk:
- A testing problem the team aims to solve
- The technique the team is proposing
- A way of evaluation to show the proposed technique works and is competent
Team Project Deliverables
Each team should submit the following:
- the team report
the implementation: a public repository link in the report (e.g. github or bitbucket repo) The team report should include:
- a precise description of the problem you attempted to solve
- a clear description of how you tried to solve the problem
- a result of experimental comparison of before and after: in other words, what benefits did your solution bring?
Additionally, each individual member should submit a separate individual report via KLMS:
- details of what you have contributed to the project
- peer assessment of your team members (yourself not included): use the scale of 10 to evaluate each of your teammates, and write clear justification for your score.
The submission deadline is 22th June, 6pm, GMT+9. The following is a submission checklist:
- Make sure you have put your group report (as pdf) and your presentation slides (as pdf) in your public project repository; make sure that we would be able to easily find it (i.e., don’t hide it in an obscure folder).
- Make sure one student from your team has submitted the repo link (plain text) on KLMS.
- Make sure you have submitted your individual report (as pdf) on KLMS.
The final presentation dates for teams have been announced in the schedule section. Each team will have up to 15 minutes. If your team is scheduled on the early date, you can just report the progress up to that point, with a clear plan for the remaining work.
Form your teams by 6th April - write down the member names in the Google Sheet document (link will be available from the Slack workspace).
Examples from the previous years
I’ve picked a few projects from 2019 that I thought was interesting below.
J. Liang, S. Elbaum, and G. Rothermel. Redefining prioritization: Continuous prioritization for continuous integration. In Proceedings of the 40th International Conference on Software Engineering, ICSE ’18, pages 688–698, New York, NY, USA, 2018. ACM.
M. Harman and P. McMinn. A theoretical and empirical analysis of evolutionary testing and hill climbing for structural test data generation. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA 2007), pages pp. 73–83. ACM Press, July 2007.
K. Pei, Y. Cao, J. Yang, and S. Jana. DeepXplore: Automated whitebox testing of deep learning systems. In Proceedings of the 26th Symposium on Operating Systems Principles, SOSP ’17, pages 1–18, New York, NY, USA, 2017. ACM.
Q. Zhu, A, Panichella, and A. Zaidman. An Investigation of Compression Techniques to Speed up Mutation Testing. In 2018 IEEE International Conference on Software Testinv, Validation, and Verification (ICST 2018), to appear.
J. Bell, O. Legunsen, M. Hilton, L. Eloussi, T. Yung, D. Marinov. DeFlaker: Automatically Detecting Flaky Tests. In 2018 International Conference on Software Engineering (ICSE 2018)
A. Amar and P. Rigby, Mining Historical Test Logs to Predict Bugs and Localize Faults in the Test Logs. In 2019 International Conference on Software Engineering (ICSE 2019), to appear.
T. Gu, C. Sun, X. Ma, C. Cao, C. Xu, Y. Yang, Q. Zhang, J. Lu, and Z. Su, Practical GUI Testing of Android Applications via Model Abstraction and Refinement. In 2019 International Conference on Software Engineering (ICSE 2019), to appear.
M. Fazzini, M. Prammer, M. d’Amorim, and A. Orso. Automatically translating bug reports into test cases for mobile apps. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2018, pages 141–152, New York, NY, USA, 2018. ACM.
M. M. Almasi, H. Hemmati, G. Fraser, P. McMinn, and J. Benefelds. Search-based detection of deviation failures in the migration of legacy spreadsheet applications. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2018, pages 266–275, 2018. ACM.
M. Kim, S.-C. Cheung, and S. Kim. Which generated test failures are fault revealing? Prioritizing failures based on inferred precondition violations using PAF. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2018, pages 679–690, New York, NY, USA, 2018. ACM.
J. Kim, R. Feldt, and S. Yoo. Guiding deep learning system testing using surprise adequacy. In Proceedings of the 41th International Conference on Software Engineering, ICSE 2019, 2019.