Prototyping, Evaluation, and Documentation
This week’s structure
This weeks curriculum has tree phases:
- Feedback on last weeks exercises
- In-class work Monday
- Homework
- Follow-up next Monday
Learning goals for the week.
After this week, you should be able to:
- Explain what technical prototypes are.
- Explain the purpose of technical prototypes in software engineering and when to use them.
- Plan, setup, and execute small experiments, such as benchmarks, to investigate certain questions with the help of the prototype.
- Formulate hypothesis, to test with a small experiment and prototype.
-
Reason about the outcomes of such experiments and argue about the quality and significance of the gathered data and results.
- Identify properties of good software documentation.
- Write appropriate technical documentation so that other people can work with your software product.
Prototyping
What is a Prototype?
Oxford Dictionary of English
A prototype is an initial version of a software system that is used to demonstrate concepts, try out design options, and find out more about the problem and its possible solutions.
Ian Sommerville, “Software Engineering 9th Edition”
In general, prototypes do not have to be executable. For example, in UI design and development paper-based mock-ups are often used to evaluate usage scenarios. They are extremely cheap to develop and setup, see Marc Rettig, “Practical Programmer: Prototyping for Tiny Fingers”. Comm. ACM, 37 (4), 21–7 (http://berea.mobi/courses/ba4abw-sp13/media/prototyping-for-tiny-fingers.pdf)
However, we focus on technical prototypes. That is executable pieces of software, which we want to evaluate.
Recap - The Architecture of Investigation
When to build a prototype?
- For example, after having performed a literature study as described in the last session but we cannot find a source that answers our question directly.
- We want to test a hypothesis or problem statement which is specific to our domain and use cases and it has not been answered before.
- When we want to execute an experiment with it and we want to have control over the situation and want to manipulate behavior directly, precisely and systematically.
What is an experiment in software engineering?
Oxford Dictionary of English
Experiments are launched when we want control over the situation and want to manipulate behavior directly, precisely and systematically. For example, if it is possible to control who is using one method and who is using another method, and when and where they are used, it is possible to perform an experiment.
Experiments may be human-oriented or technology-oriented. In human-oriented experiments, humans apply different treatments to objects, for example, two inspection methods are applied to two pieces of code. In technology-oriented experiments, typically different tools are applied to different objects, for example, two test case generation tools are applied to the same programs.
Claes Wohlin, et al. “Experimentation in Software Engineering”
Experiments are appropriate to investigate different aspects, including:
- Confirm theories, i.e. to test existing theories.
- Confirm conventional wisdom, i.e. to test people’s conceptions. … The strength of an experiment is that it can investigate in which situations the claims are true and they can provide a context in which certain standards, methods and tools are recommended for use.
Claes Wohlin, et al. “Experimentation in Software Engineering”
Benchmarking examples
Oxford Dictionary of English
Benchmarks are just a certain type of experiments in which you compare different but similar solutions or prototypes to each other.
Your turn!
Read the following blog posts, which compare various web frameworks with each other.
- https://medium.com/@tschundeee/express-vs-flask-vs-go-acc0879c2122
- https://klen.github.io/py-frameworks-bench/#methodic
- http://nichol.as/benchmark-of-python-web-servers
- https://www.codementor.io/garethdwyer/flask-vs-django-why-flask-might-be-better-4xs7mdf8v
- https://stackshare.io/stackups/django-rest-framework-vs-expressjs-vs-flask
- https://medium.com/entrepreneurguide-me/rails-vs-django-vs-laravel-an-analysis-of-web-frameworks-from-a-non-technical-founder-312506969afd
Describe with a few bullet points
- What is the question that the authors try to answer?
- What is their hypothesis/problem statement?
- Do they present an experiment with one or more prototypes?
- If yes: what are the prototypes?
- If not: what is missing to make it an experiment?
- Are there experiments reproducible?
- If so, can you try to reproduce some of their results?
- Are the presented results understandable?
- If not: why not? What is missing?
- How convincing are the drawn conclusions?
- If they are not convincing: what is missing? What would make them more convincing?
Exercise on Benchmarking… due next week
There are three webservers running in different parts of the world. They serve a static “Hello, World!”” message. You can access them under the following addresses:
Your task is to plan, setup, execute, and evaluate a small experiment concerning these three webservers. In particluar you have to:
- Formulate a hypothesis/problem statement about behavior of response times of these three web servers.
- Plan an experiment, which measures response times of these three web servers.
- Execute the experiment, which measures response times of these three webservers.
- Evaluate your experiment and interpret the measurements and results.
- Discuss what you are measuring, how you are measuring, and what could influence your results., see Peter Sestoft “Microbenchmarks in Java and C#” https://www.itu.dk/people/sestoft/papers/benchmarking.pdf for inspiration
OBS you should execute the experiment twice. Once you measure response times from school and once from home.
Write a document that explains all of the above. That is, that formulates your hypothesis, that describes the experiment setup, that describes how you executed the experiment, that explains how to reproduce your experiment, and that collects, interprets, and discusses you measurement results.