Observation, Surveys/Questionnaires, and Interviews
This week’s structure
This weeks curriculum has tree phases:
- Feedback on last weeks exercises
- In-class work Monday
- Homework
- Follow-up next Monday
Learning goals for the week.
After this week, you should be able to:
- Explain the purpose of user observations in software engineering and when to use them.
- Execute and evaluate small experiments with user observations.
- Explain the purpose of surveys with questionnaires and interviews in software engineering and when to use them.
- Execute and evaluate small surveys.
Observation
-
“Observations can be conducted in order to investigate how a certain task is conducted by [subjects].” Per Runeson and Martin Höst, Guidelines for conducting and reporting case study research in software engineering
-
“An advantage of observations is that they may provide a deep understanding of the phenomenon that is studied. Further, it is particularly relevant to use observations, where it is suspected that there is a deviation between an “official” view of matters and the “real” case (Robinson et al. 2007). It should however be noted that it produces a substantial amount of data which makes the analysis time consuming.” Per Runeson and Martin Höst, Guidelines for conducting and reporting case study research in software engineering
Your turn!
Observation
Imagine you are a student working for the company devopsgroup.io. During your internship you built a tool that allows to automate the administration of Digital Ocean droplets via Vagrantfiles.
Now, we want to evaluate if the newly created tool is usable in a real-world setup and that it’s documentation is understandable and useful. To do so, we decide to setup a small experiment in which we let other developers use the new tool.
For this task this new tool is the vagrant-digitalocean Vagrant provider, see https://github.com/devopsgroup-io/vagrant-digitalocean.
The task for the experiment subject
- Navigate to https://github.com/devopsgroup-io/vagrant-digitalocean.
- Install the
vagrant-digitaloceanVagrant provider and configure it as described on the page. - Now, create a
Vagrantfilethat creates two droplets at Digitalocean.- Choose the smallest ones with 512mb.
- Let one droplet be a 64-bit Ubuntu Linux 16.04 and the other one a 64-bit FreeBSD 11.1.
- Startup the machines.
- Find the IP of the FreeBSD machine.
- Ping the FreeBSD machine from the Ubuntu machine from your local host machine by running
vagrant ssh -c 'ping <IP_of_FreeBSD_machine>'- Can you establish a connection between the two machines?
- Destroy the droplets from the command-line as described on https://github.com/devopsgroup-io/vagrant-digitalocean
The task for the observers
- Install a screen capture tool, i.e., a tool recording screen contents onto the machine on which the experiment subject will work.
- See https://lifehacker.com/5880928/the-best-screen-capture-tool-for-windows for a list of tools for Windows.
- On OS X you can use the pre-installed Quicktime.
- Start a recording and subsequently, let her start working on the task described above.
- While the experiment subject is performing the task, observe what he/she is doing and take notes/ keep a log on everything that you deem important, such as:
- Which programs and tools are used and to complete the task?
- On what is the experiment subject spending time and how long?
- Which problems do you observe?
- OBS do neither intervene while the person is working on the task nor provide any help.
Reflection
- What did you observe?
- What did experiment subject do to complete the task?
- How long time did the experiment subject spent on reading documentation in total?
- Which parts of the documentation were longest under focus?
- How long did it take to create the corresponding
Vagrantfilein total? - Did the experiment subject experience any problems?
- On which part of the documentation or while using which part of the vagrant tool?
-
When in doubt check the recording again.
- What is your conclusion concerning the usability of the tool and the quality of the documentation?
Surveys with Questionnaires
A survey is the “collection of standardized information from a specific population, or some sample from one, usually, but not necessarily by means of a questionnaire or interview” Colin Robson, Real World Research
How to design a survey?
What is a Likert scale?
Open and closed questions?
http://polaris.gseis.ucla.edu/jrichardson/dis220/openclosed.htm
Your turn!
Design and create a questionnaire, to evaluate the usability and the quality of the PMD tool from the last session. Focus for example on how hard it is to create reports for certain metrics on a Java program, how helpful or clear the metric results are described and presented, and try to figure out what your respondents would suggest as possible improvements to the tool.
- Create the questionnaire with Google Forms, see one of the following tutorials:
- Let at least three person from another group reply to the questionnaire.
- Evaluate the collected results.
- Try to condense which opinions about PMD the the three respondents share and in which they disagree.
- Try to formulate a suggestion for possible improvements to the PMD tool out of the collected replies.
Homework until in two weeks
- Prepare a questionnaire with Google Forms to get feedback on the quality of your operation documentation in LSD.
- Let your questionnaire consist of at maximum 15 items.
- Choose those items, that deem to be most suitable for this task.
- Send the questionnaire to the team, which is operating your Hacker News Clone project.
- Give the responding team two days to fill out the questionnaire.
- Evaluate the responses that you collected from them.
- In case you realize that the responding team points out issues with your current operation documentation, modify and improve it accordingly.
- Inform the operations team that an improved version of the operations documentation is accessible and point out what you have changed, i.e., provide a changelog.