diff --git a/doc/informal.tex b/doc/informal.tex index 2f24fdcb..16b217e8 100644 --- a/doc/informal.tex +++ b/doc/informal.tex @@ -291,6 +291,11 @@ def testBench(width): \end{verbatim} +Note how we use the conversion function \code{bin} to get a binary +string representation of the signal values. This function is exported +by the \code{myhdl} packages and complements the standard Python +\code{hex} and \code{oct} conversion functions. + To demonstrate, we set up a simulation for a small width: \begin{verbatim} diff --git a/doc/unittest.tex b/doc/unittest.tex new file mode 100644 index 00000000..8e9a656a --- /dev/null +++ b/doc/unittest.tex @@ -0,0 +1,394 @@ +\chapter{Unit testing} + +\section{Introduction} + +It is the present author's opinion that many aspects in the design +flow of modern digital hardware design can be viewed as a special kind +of software development. From that viewpoint, it is a natural question +whether advances in software design techniques can not also be applied +to hardware design. + +One software design approach that is in the news lately is +\emph{Extreme Programming} (XP). It is a fascinating set of techniques and +guidelines that often seems to go against the conventional wisdom. On +other occasions, XP just seems to emphasize the common sense, which +doesn't always coincide with common practice. For example, XP stresses +the importance of normal, non-overtime workweeks, if we are to have the +fresh mind needed for good software development. Refreshing to hear, +if you ask me, although hardly surprizing. + +It is not my intention nor qualification to present a tutorial on +Extreme Programming. Instead, in this section I will highlight one XP +concept which I think is very relevant to hardware design: the +importance and methodology of unit testing. + +\section{The importance of unit tests} + +Unit testing is one of the corner stones of Extreme Programming. Other +XP concepts, such as collective ownership of code and continuous +refinement, are only possible by having unit tests. Moreover, XP +emphasizes that writing unit tests should be automated, that they should +test everything in every class, and that they should run perfectly all +the time. + +I believe that these concepts apply directly to hardware design. In +addition, unit tests are a way to manage simulation time. For example, +a state machine that runs very slowly on infrequent events may be +impossible to verify at the system level, even on the fastest +simulator. On the other hand, it may be easy to verify it exhaustively +in a unit test, even on the slowest simulator. + +It is clear that unit tests have compelling advantages. On the other +hand, if we need to test everything, we are going to have to write +lots and lots of unit tests. So it will have to be easy and pleasant +to create, manage and run them. Therefore, XP emphasizes the need for +a unit test framework to support these tasks. In the following, +we will explore the use of the \code{unittest} module from +the standard Python library for creating unit tests for hardware +designs. + + +\section{Unit test development} + +In this section, we will informally explore the application of unit +test techniques to hardware design. We will do so by a (small) +example: testing a binary to Gray encoder as introduced in +section~\ref{gray}. + +\subsection{Defining the requirements} + +We start by defining our requirements. For a Gray encoder, we want to +the output to comply with Gray code characteristics. Let's define a +\dfn{code} as a list of \dfn{codewords}, where a codeword is a bit +string. A code of order \code{n} has \code{2**n} codewords. + +A well-known characteristic is the one that Gray codes are all about: + +\newtheorem{reqGray}{Requirement} +\begin{reqGray} +Consecutive codewords in a Gray code should differ in a single bit. +\end{reqGray} + +Is this sufficient? Not quite: suppose for example that an +implementation returns the lsb of each binary input. This would comply +with the requirement, but is obviously not wat we want. Also, we don't +want the bit width of Gray codewords to exceed the bit width of the +binary codewords. + +\begin{reqGray} +Each codeword in a Gray code of order n must occur exactly once in the +binary code of the same order. +\end{reqGray} + +With the requirements written down we can proceed. + +\subsection{Writing the test first} + +A very fascinating guideline in the XP world is to write the unit test +first. That is, before implementing something, first write the test +that will verify it. This seems to go against our natural inclination, +and certainly agains common practices. Many engineers like to +implement first and think about verification afterwards. + +But if you think about it, it makes a lot of sense to deal +with verification first. Verification is about the requirements only +--- so your thoughts are not yet cluttered with implementation +details. The unit tests are an exectutable description of the +requirements, so it will be very clear what needs to be done and they +will be better understood. It follows that the implementation should +go smoother. Perhaps most importantly, the test is available when you +are done implementing and can be run at any time by anybody to verify +changes by anybody else. + +Python has a standard \code{unittest} module to facilitate writing, +managing and running unit tests. With \code{unittest}, test case are +written by creating a class that inherits from +\code{unittest.TestCase}. Individual tests are created by methods of +that class: all methods that start with \code{test} are considered to +be tests of the test case. + +We will define a test case for the Gray code properties, and then +write a test for each of the requirements. The outline of the test +case class is as follows: + +\begin{verbatim} +from unittest import TestCase + +class TestGrayCodeProperties(TestCase): + + def testSingleBitChange(self): + """ Check that only one bit changes in successive codewords """ + .... + + + def testUniqueCodeWords(self): + """ Check that all codewords occur exactly once """ + .... + +\end{verbatim} + +Each method will be a small test bench that tests a single +requirement. To write the tests, we don't need an implementation of +the Gray encoder, but we do need the interface of the design. We can +specify this by a dummy implementation, as follows: + +\begin{verbatim} +def bin2gray(B, G, width): + ### NOT IMPLEMENTED YET! ### + yield None + +\end{verbatim} + +For the first requirement, we will write a testbench that applies all +consecutive input numbers, and compares the current output with the +previous one for each input. Then we check that the difference is a +single bit. We will test all Gray codes up to a certain order +\code{MAX_WIDTH}. + +\begin{verbatim} + def testSingleBitChange(self): + + """ Check that only one bit changes in successive codewords """ + + B = Signal(intbv(-1)) + G = Signal(intbv(0)) + G_Z = Signal(intbv(0)) + + def test(width): + B.next = intbv(0) + yield delay(10) + for i in range(1, 2**width): + G_Z.next = G + B.next = intbv(i) + yield delay(10) + diffcode = bin(G ^ G_Z) + self.assertEqual(diffcode.count('1'), 1) + + for width in range(MAX_WIDTH): + dut = bin2gray(B, G, width) + sim = Simulation(dut, test(width)) + sim.run(quiet=1) + +\end{verbatim} +Note how the actual check is performed by a \code{self.assertEqual} +method, defined by the \code{unittest.TestCase} class. + +Similarly, we write a test bench for the second requirement. Again, we +simulate all numbers, and put the result in a list. The requirement +implies that if we sort the result list, we should get a range of +numbers: + +\begin{verbatim} + def testUniqueCodeWords(self): + + """ Check that all codewords occur exactly once """ + + B = Signal(intbv(-1)) + G = Signal(intbv(0)) + + def test(width): + actual = [] + for i in range(2**width): + B.next = intbv(i) + yield delay(10) + actual.append(int(G)) + actual.sort() + expected = range(2**width) + self.assertEqual(actual, expected) + + for width in range(MAX_WIDTH): + dut = bin2gray(B, G, width) + sim = Simulation(dut, test(width)) + sim.run(quiet=1) + +\end{verbatim} + + +\subsection{Test-driven implementation} + +With the test written, we begin with the implementation. For +illutration purposes, we will intentionally write some incorrect +implementations to see how the test behaves. + +The easiest way to run tests defined with the \code{unittest} +framework, is to put a call to its \code{main} method at the end of +the test module: + +\begin{verbatim} +unittest.main() + +\end{verbatim} + +Let's run the test using the dummy Gray encoder shown earlier: + +\begin{verbatim} +% python test_gray.py -v +Check that only one bit changes in successive codewords ... FAIL +Check that all codewords occur exactly once ... FAIL + + +\end{verbatim} + +As expected, this fails completely. Let's now try an incorrect +implementation, that puts the lsb of in the input on the output: + +\begin{verbatim} +def bin2gray(B, G, width): + ### INCORRECT - DEMO PURPOSE ONLY! ### + while 1: + yield B + G.next = B[0] + +\end{verbatim} + + +Running the test produces: + +\begin{verbatim} +% python test_gray.py -v +Check that only one bit changes in successive codewords ... ok +Check that all codewords occur exactly once ... FAIL + +====================================================================== +FAIL: Check that all codewords occur exactly once +---------------------------------------------------------------------- +Traceback (most recent call last): + File "test_gray.py", line 109, in testUniqueCodeWords + sim.run(quiet=1) + File "/home/jand/project/myhdl/myhdl/Simulation.py", line 87, in run + clauses, clone = waiter.next() + File "/home/jand/project/myhdl/myhdl/Simulation.py", line 161, in next + clause = self.generator.next() + File "test_gray.py", line 104, in test + self.assertEqual(actual, expected) + File "/usr/local/lib/python2.2/unittest.py", line 286, in failUnlessEqual + raise self.failureException, \ +AssertionError: [0, 0, 1, 1] != [0, 1, 2, 3] + +---------------------------------------------------------------------- +Ran 2 tests in 0.785s + +\end{verbatim} + +Now the test passes the first requirement, as expected, but fails the +second one. After the test feedback, a full traceback is shown that +can help to debug the test output. + +Finally, if we use the correct implementation as in +section~\ref{gray}, the output is: + +\begin{verbatim} +% python test_gray.py -v +Check that only one bit changes in successive codewords ... ok +Check that all codewords occur exactly once ... ok + +---------------------------------------------------------------------- +Ran 2 tests in 6.364s + +OK + +\end{verbatim} + + + +\subsection{Changing requirements} + +In the previous section, we concentrated on the general requirements +of a Gray code. It is possible to specify these without specifying the +actual code. It is easy to see that there are several codes +that satisfy these requirements. In good XP style, we only tested for +the requirements and nothing more. + +It may be that more control is needed. For example, the requirement +may be for a particular code, instead of compliance with general +properties. As an illustration, we will show how to test for +\emph{the} original Gray code, which is one specific instance that +satifies the requirements of the previous section. In this particular +case, this test will actually be easier than the previous one. + +We denote the original Gray code of order \code{n} as \code{Ln}. Some +examples: + +\begin{verbatim} +L1 = ['0', '1'] +L2 = ['00', '01', '11', '10'] +L3 = ['000', '001', '011', '010', '110', '111', '101', 100'] + +\end{verbatim} + +It is possible to specify these codes by a recursive algorithm, as +follows: + +\begin{enumerate} +\item L1 = ['0', '1'] +\item Ln+1 can be obtained from Ln as follows. Create a new code Ln0 by +prefixing all codewords of Ln with '0'. Create another new code Ln1 by +prefixing all codewords of Ln with '1', and reversing their +order. Ln+1 is the concatenation of Ln0 and Ln1. +\end{enumerate} + +Python is well-known for its support for elegant algorithmic +descriptions, and this is a good example. We can write the algorithm +in Python as follows: + +\begin{verbatim} +def nextLn(Ln): + """ Return Gray code Ln+1, given Ln. """ + Ln0 = ['0' + codeword for codeword in Ln] + Ln1 = ['1' + codeword for codeword in Ln] + Ln1.reverse() + return Ln0 + Ln1 + +\end{verbatim} + +The code \samp{['0' + codeword for ...]} is called a \dfn{list +comprehension}. It is a concise way to describe lists built by short +computations in a for loop. + +The requirement is now simply that the output code matches the +expected code Ln. We use the \code{nextLn} function to compute the +expected result. The new test case code is as follows: + +\begin{verbatim} +class TestOriginalGrayCode(TestCase): + + def testOriginalGrayCode(self): + + """ Check that the code is an original Gray code """ + + B = Signal(intbv(-1)) + G = Signal(intbv(0)) + Rn = [] + + def stimulus(n): + for i in range(2**n): + B.next = intbv(i) + yield delay(10) + Rn.append(bin(G, width=n)) + + Ln = ['0', '1'] # n == 1 + for n in range(2, MAX_WIDTH): + Ln = nextLn(Ln) + del Rn[:] + dut = bin2gray(B, G, n) + sim = Simulation(dut, stimulus(n)) + sim.run(quiet=1) + self.assertEqual(Ln, Rn) + +\end{verbatim} + +As it happens, our implementation is apparently an original Gray code: + +\begin{verbatim} +% python test_gray.py -v TestOriginalGrayCode +Check that the code is an original Gray code ... ok + +---------------------------------------------------------------------- +Ran 1 tests in 3.091s + +OK +\end{verbatim} + + + +