In this assignment you will use two different static analysis tools to automatically detect potential defects.
The first static analysis tool is GrammaTech's CodeSonar, which focuses on security issues, as well as memory, resource and concurrency defects. CodeSonar is a commercial tool used in activities such as DO-178B avionics certification; we have obtained an academic license for its use in this class.
The second static analysis tool is Facebook's Infer, which focuses on memory errors, leaks, race conditions, and API issues. Infer is open source.
You may work with a partner for this assignment. If you do you must use the same partner for all sub-components of this assignment. Use Gradescope's partner selection feature. Only one partner needs to submit the report on Gradescope, but if you both do, nothing fatal happens.It is your responsibility to download, compile, run and analyze the subject program and associated tools (or use the precompiled one). Getting the code and tools to work in some manner is part of the assignment. You can post on the forum for help and compare notes bemoaning various architectures (e.g., windows vs. mac vs. linux, etc.). Ultimately, however, it is your responsibility to read the documentation for these programs and tools and use some elbow grease to make them work.
We will make use of the lighttpd webserver (pronounced "lighty"), version 1.4.17, as our primary subject program for this homework. A local mirror copy of lighttpd-1.4.17.tar.gz is available, but you can also get it from the original website. It is about 55,000 lines of code in about 90 files. While somewhat small for this class, some analysis tool licenses have LOC limits or scalability issues, so it was chosen as an indicative compromise.
While not as large or popular as apache, at various points lighttpd has been used by YouTube, xkcd and Wikimedia. Much like apache, old verisons of it have a number of known security vulnerabilities.
The Common Vulnerabilities and Exposures system is one approach for tracking security vulnerabilities. A CVE is basically a formal description, prepared by security experts, of a software bug that has security implications.
There are nine CVEs associated with lighttpd 1.4.17. For example, CVE-2014-2324 has the description "Multiple directory traversal vulnerabilities in (1) mod_evhost and (2) mod_simple_vhost in lighttpd before 1.4.35 allow remote attackers to read arbitrary files via a .. (dot dot) in the host name, related to request_check_hostname." You can dig into the information listed in, or linked from, a CVE (or just look at subsequent versions of the program where the bug is fixed!) to track down details. Continuing the above example, mod_evhost refers to source file mod_evhost.c, mod_simple_vhost refers to file mod_simple_vhost.c, and request_check_hostname is in file request.c. You will need such information when evaluating the whether or not the tools find these security bugs.
The Infer tool is a static analyzer — it detects bugs in programs without running them. The primary website is fbinfer.com.
Unfortunately, some versions of Infer can be obnoxious to build and install, despite their handy installation guide.
As a conveience only (see above about "your responsibility"), a pre-compiled, runs-for-me-but-no-promises-for-you (Ubuntu 16.04.2 LTS GNU/Linux 4.4.0-34-generic x86_64) version of Infer is available locally here (warning: 265 MB). The main binary can be found at infer-linux64-v0.13.0/infer/bin/infer. You can use either the pre-compiled one or compile it yourself for full credit.
Once you have Infer built or downloaded, applying it to lighttpd should be as simple as:
$ cd lighttpd-1.4.17 $ sh configure $ /path/to/infer/bin/infer run -- make
That should produce output similar to (but everything is fine if you get very different numbers):
make[1]: Leaving directory '/home/weimer/src/lighttpd-1.4.17'
Found 88 source files to analyze in /home/weimer/src/lighttpd-1.4.17/infer-out
Starting analysis...
legend:
"F" analyzing a file
"." analyzing a procedure
FFFFFFFFFF.....F...FF....F..FF.F..F....................................................................................FF.................................................F...........F..................F..................F...........................................................................F....................................................................F........................................................F.......F.................F...............F.......FF.............F...................F.............F.........F...F.................F...................................F............FF.F.....F.......................F.....FF..............F..F........FF..........FF.............FF.......FF.F....F......F......FFF..............F.........F...F......F...........F.......FF..........F.F...........F...F..F.......F..F...F........................F..F.........F....F........F.....F..F..........F............F....F...................F................................................................................................................................................
Found 308 issues
src/joblist.c:19: error: NULL_DEREFERENCE
pointer `srv->joblist->ptr` last assigned on line 16 could be null and is dereferenced at line 19, column 2.
17. }
18.
19. > srv->joblist->ptr[srv->joblist->used++] = con;
20.
21. return 0;
...
Summary of the reports
NULL_DEREFERENCE: 145
DEAD_STORE: 94
MEMORY_LEAK: 65
RESOURCE_LEAK: 3
QUANDARY_TAINT_ERROR: 1
You will have to read through the output carefully and analyze the reported defects. Some will be true positives (i.e., real bugs in the code) and some will be false positives (i.e., spurious warnings that do not correspond to real bugs).
Running Infer on jfreechart-1.5.0 is similarly direct.
$ cd jfreechart-1.5.0-1.4.17
$ /path/to/infer/bin/infer run -- mvn compile
Capturing in maven mode...
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building JFreeChart 1.5.0
...
Found 640 source files to analyze in /home/weimer/src/jfreechart-1.5.0/infer-out
Starting analysis...
...
Found 69 issues
src/main/java/org/jfree/data/xml/DatasetReader.java:73: error: RESOURCE_LEAK
resource of type `java.io.FileInputStream` acquired to `in` by call to `FileInputStream(...)` at line 72 is not released after line 73.
71. throws IOException {
72. InputStream in = new FileInputStream(file);
73. > return readPieDatasetFromXML(in);
74. }
...
Summary of the reports
THREAD_SAFETY_VIOLATION: 43
NULL_DEREFERENCE: 22
RESOURCE_LEAK: 4
GrammaTech's CodeSonar static analyzer is a commercial (not open source) tool for finding defects in program source code or binaries.
GrammaTech has generously provided an academic license so that students in this class can make use of their tool on a limited basis. This license includes a lines-of-code limit, so we have pre-run the analysis and made the results available for everyone to share (running it is very similar to Infer, see below). Analyzing additional code or trying to subvert this license runs the risk of ruining our relationship with that company and thus preventing me from giving future students this experience in subsequent semesters — doing so is thus a significant academic integrity violation.
CodeSonar's output is designed to be shared among an organization's developers. As a result, the analysis is carried out once and then the reports are made available to everyone via a web interface. In this model the team might work together to triage and prioritize the defect reports, assigning some to one developer and some to another. We have already run CodeSonar for you; all students share the analysis results.
Running CodeSonar is as simple as running Infer — basically, instead of running make one runs codesonar make. The actual commands are listed below, but you do not run them (we have already run them for you):
$ cd lighttpd-1.4.17 $ sh configure $ # DO NOT: codesonar analyze lighttpd-1.4.17 host:port make
Similarly, a Java project would be analyzed by (do not run this):
$ cd jfreechart-1.5.0/ $ mvn compile $ # DO NOT: codesonar analyze jfreechart-1.5.0 host:port cs-java-scan src/main/java/
The CodeSonar reports can be found at this location. If you are registered with the course (e.g., via Wolverine Access), click "Forgot Password", enter your UM username, and click the Send Email button. Login information will be emailed to your @umich.edu address.
For example, your instructor would enter weimerw (note: no @ or numbers typed in this form field) to have an email with a set-your-password login code sent to weimerw@umich.edu.
(The license agreement places certain restrictions on who can use or see the service.)
We also make available the CodeSonar analyses of "fan favorites" such as:
Note that the report requires you to choose an additional program (such as one of the four listed above) and analyze it.
You must write a detailed PDF report reflecting on your experiences with these static analysis defect detection tools. In particular, all of the following are required:
The grading staff will select a small number of excerpts from particularly high-quality or instructive reports and share them with the class. If your report is selected you will receive extra credit.
Submit a single PDF report via Gradescope. You must include your name and UM email id (as well as your partner's name and email id, if applicable).
There is no explicit format (e.g., for headings or citations) required. For example, you may either use an essay structure or a point-by-point list of question answers.