ࡱ> n w:Ffnrر;lOzrPNG  IHDRXGVPLTE>ȮbKGDHIDATxb:EabgNse-6et `jg.W@2zaA AX ,H$  ‚aA AX ,H$ [] _"_u^xu8+YP W. ] +(OK.~$6aaoz;JXX{/PNXWBhoo}a}_DX?*qCqׄ@ZS冄=MӴT}  ‚aA AXh k3:<}b_v>fxOokz t(TN|e7i&S,zay kzne)=šFj j"S=sv}œR!巊_hu8j^|{"U4ne@k5evW]i+r竏`"kgIw+^aWT޻,+x+aE*K"OkoT9jݮ;_G|\udfnU{.*zŗMaz]F$Oh]᫬Šn_B8[V~;aa-3dX))af5 k+xTlfLT޿EcXk +x\n(xC/S/aUlݮ˰GT\x}Q|#VSkaǼ&(~+G!ίx\ο8He1XW9*oϞʒ}]|g ^֯FS@ AX ,H$  ‚aA AX ,H$.DX ,H$  ‚aA AX ,H$  ‚aA AX ,H$  ‚aA AX ,H$  ‚aA AX ,H$  ‚D)/=..7ʎ['8/[h' zDr(kVܗ['֕-GY5GK*}vsqXk vf{֯;w1:vo< aؚ1G![>9(բckxPjAXaՏHJgaFX[0BX:a|2BXR !,_)CFW!#+Őba5G>#,_5 a)aI l@X%qUY蘆AX 7Eh0=gHo\T >?k, +>~$a)uirM@X%AX%qa}^wHl!,)CF:p?aJ1d|2BXR !,_)CFW!#DR 9CXSHU!#s,J1da fx"U)T2Ҿȇ^'F-DR i^dǮx&5).eWʲ<CFٳ=eyZ!#mlbLOf%ŐEe_qƃFCF,#\a}A"܈a9=bH"Q aGv7]7.Ю:ꘓ`ovVϜ ;7j1pP)l/늰aswVgW [<ZmR Xd^vip^MR ١ְ>zP)lUcX=CFEJiǩK1d$Xdz~H1dd#RAXY Ҫ.Umג *|! j>!V|ke +rU.kOXYr\֞Z,5VVEtY{jȗQ#VV'+ū+:kgCFaU/r.kOXYy.kOXY[RC>௰ˆ1D3~4lC>%ˆ1D3^ʺlC>-ˆ1D3veC]"粡.s ̃}sP9H}.2g []URˡؒU}V.~: 8ig(ԨbԞp-lmtIci,9~Wk}r6YL*_`7Q"0yX~[.>Wn]GG!5RZ`z+oJ߻PNG  IHDRXGVPLTE3OehbKGDHIDATx흋(E̍ E)sZV@$HL .@,.@,.@,.@,.@,.@,.@,]QsvEm"@_e"ٕ-]"Y(z53_H0 P$)X'4 Ȋ_Ē@XsX@sMb+b+?rWtdƱpYX 9Gf#!{yf3] 5!+    '(]䳨 Xsl``ŚgԊU<ìW,-6FUߚD˯;EZ}i}5*9{.EϴulfkF+%x+bS2SJd?UsG4XJnea bp>_XqazJ#dFf$ڢ^Z_y% no7U\bu@KSwڬ+׫Bh^cь5k`>շXR!VиjOzak۴UNd-hFz!jRXOoX[ǚ*mJi'>D3}1zp,[69 ctu!{렫V pb pb pb pb pb pb pB,x5(}|C Pa:Ǚ+9so2x@WԊᓾ,ÐWiBw%#>UuJvhUjE&VnPk|R"=*L5>bWZX/>zNů g"VU—qƢOYu[J $}G `Agnt@,mbUpb pb pb pb pb pb pby{j,gմ'ɼ3s{vUM!/m Rɪ*[Kj=w}9YOaK~Lr[#roKx|NH l3ߘ~S{/oϳ8*e>XelrV *S9˘zdꪑ̪>Z7X$"ľK&놉0 PCG`hiCZuX-+''cX 2,?;bq|WR'J[X:-KV&Zͪ帥Wt}9"$bi~A,4Z,v4P-wm5)IzK~5 A SۦtUXJnb:b51i&0XijMXǃ5vDZJAfS>:3 Yެ]3MXRSx0ü,*m"~+n,'yJE\_ Xb==Xx*yuXGvk~JV}+58I^7:\~ė3xB,o΋Fb pb pb pb pb pbΈuƕ\.{NsW|ShM G"[†Ma&ծXX^+V,ڿŚb@X9-ωځ^);+߸*n ϏcJ7 w ]tS{T= pb V<1:i4z #z{G% K-ۈ~Aeitt`?7MDVWvtN*X$_M'bε|S[]ŊMOLvZ&O{D2OVb]n1ծhb!?~vi7*gQ7SbOJ!"}'b χ;T.֪b&b?XZ_nix*gW_&VirQxXĩ\S(6 5?’P(ƺ(k 45]w "6.ּRf,6Yxk4Y-B)K4J7|X12bY,#ةdCر],#$c5M(5XbnkBݝg-?<gD%XyW˕ ;^>VbyX uJlyWXj8V_ 6\ܡ4XzY.V7%zM[xD^7+ïuY)eeW4+!RjA -v͒bzc;~!:bx q&1Uز1$% VHH%ǗW*#i VyyCץvI>!K'q7Ԭ͔ D!/cˤX-5+_"G[=)iy>kSKbbɊb)k /򜚶4$>VXY: jf8KuUM!:6'쒶eY! *dm'Vrt(mt<%>mrGĉ<WGbY Ӗd q~JiK8?"VW#=g`Lˤgrż.Tp8 u܉ܦ6/+|Jjyz.!@ZGQݲvh'iԐbbɺIg2Xc1m_; 㒙2Wy*1u2bM GlېvoK6 )mpsXgu'HvwM&͒iLJ^mq=q55dE"&)|B#I~z^,V2G vϩO3:S r0"_X2GnyFR,֑`֥ZJ:!Vlyhxr$X$4x.K8BIyeNךA&UF,_s9( #h+I,:%,% xнb 35bIX>w_d ҪѐP Y'cUl /V F1HM$nqbxW\udZ#th+ }.DZT7|"T Lip"I͖uqo%SVJ ó=9̵ɇXvx͏ЏW+܍PpVR?ʜQKٿRvwm3{i?ws)|W7V罂X.+ .d~f*4kOr?@ 2W 2J+av7g>.oIc`TlHF#&QQ$egB}XnY>' Ȳ*˓tljbq/2IÂR$ ZQf}'-i/"eh-k,bN,^툥b *juA6e$.MuΜY^2HJ:*x8d )B?/b$i;R!O8 *^+H[{N"ˆXZP?D9.D2b4YRCCnbJO1: 5@@S|ϰ(#RX׬{}"3y)?'ΰjï3^/K#e ^+!AU^XxX>zʺb]b2ɻIs ~b9O7 bZH^bA,|rVD ("Vٖ| tAeYW^@uW~O dJM'u׊EFEQ?;,zb]YFC_t+X% 7/n K@,o u-We 7wcƱqJ(hHR pb pb pb pb pb pb pbڋK?9`.{§=~b^̨ˤfO095Y.; q#2BALJ,>-/~$՜4dUXz Lvm)xY1O1 `VkNŌc(nuD})hFb[z˓! iXb~1(nR*$u\b-IPXtO|!k95nv6++K4I,!˖qS<'fe@b&%EՓX0^*w6Xw-w&@Iͯ,eڧX ȞN(]8{9g VXKw+Es,DZ)r]̎cBX\A b*d5/ K,{%.uHH$9[V+L6ץ y̜7IJK2Y%WxKԲJNqR!C{É}N%tR7h I ‚`}ĉAɉe¨ҘVufVÈUmlҚ4Фe/3+!k L_k l`O,e$ئMR[MxA&-Id6`nҾX *vb͂XZuCl3%EJW V'ЏzWA"bqK22`(}K K(!j }]Ҭ+5X.%*L]c勘+%XAݝmI3۩XTU*+p}l@,W{m^5>>]@q*˓@,.@,.@,.@,.@,.@,9lIENDB`n6 &0t+FjuB &)YXPNG  IHDRXGVPLTE>ȮbKGDH{IDATxb:Fa {v×7 ?#ªZۙykpO ‚aA AX ,H$  ‚aA AX ,Hs-DKr!zHxa=;-\l?"a=J^ ,,=h!p+l%,D(|R DBF2'0a=鰖;BX k%,$y9 VE5c7oOOcCBeYOݕ{B~V  ‚aA AX*%ES/*%\1*/ZvvR0rvbv'~ϙϢYVMwo4òl5?#+f_&o8j)*.px1}ѢxgMjlƌ˜\ý_$+2E[׹ram%VK!kCX ,H$  ‚aA p{=$  ‚aA AX ,H$  ‚aA AX ,H$  ‚aA AX ,H$  ‚aA AX ,HԄO[uv>4nz°#$JaBt.۞AXj ~I 8’;ڮoeMgaྒmyx c :HmS78qX~K:~w/gk?]zp $ؽ):vuBg#z_.Tx`8lXZKoo尜c8+ïjGXJsK?!E p^Xnx lTXEX~ {2~ <(xe*ǜ ˿q0|uigx fzU!uؙy_J&aE26ܰwUZF92Kw!M?#o6Cf+"'x6q_ܭ]ޞp aůIQ.+S&,rx|𻱻gևe:oZ}`[u#tqEk:"5{\šW ‚aA AX ,H$  ‚aA AX ,H$  ‚aA AX ,H$  ‚aA AX ,H$  ‚aA AX ,H$  ‚aA AX ,H$(,׫?*OjO=rX~|i6J{iQ߽-ga=)wUrySOHۄx)S}hY;FJ?+$,Q!+utx Q &jZp˅|/h_>aa%,h^ {  1jg][sDV{ĥ"؋{\YuEX+/eGX*d.KJX23FX:ad%4kX=',9LJXJSaIMXV!taJf+xKln!,7{5?2QX#f[nMSV';U=bu k|aXo3չVʦ̵_Q~$.:(w+`.옰˻qX{~սkX{2濉=#q̛54OX?C.'ۆ܌?so-OT?Э36)kԧYP){XW9°&൮&S!+$ѧa) *u?.a! *Vg,SQ?.RhLX#"Nz冥Vj-OfY8W5f3Z V8)q-*YmFqJ7 .wkk_<} 3šEݶN܏1 atVU1 kmO焵f>HYo/VbB{+Pew_Z6Κ7_䄸[ziV s湲f̄xeO{iV*7]=Kt&*5UgLd3wReX3x~taw~1nBuXh&EX-RsT^=5RLS1mO_e&q_MH_Xkçf+t)~W8ĝoª+&ªtf`LV}VVݏt> ~a9?zXÂw^}:%q{{:y :t3~m-„Ԟ3ӿ!,H$  ‚aA AX ,H$  ‚aA AX ,H$  ‚aA AX ,H$  ‚aA AX ,H$  ‚aA AX ,H$  ‚aA AX ,H$  ‚aA AX ,H$  ‚aA AX ,H$  ‚aA AX ,H$  ‚aA AX ,H$  ‚aA AX ,H$  ‚aA AX ,H$  ‚aA AX ,H$  ‚aA AX ,H$  ‚aA AX ,H$  ‚aA AX ,H$  ‚aA AX ,H$  ‚aA AX ,H$  ‚aA AX ,H$  ‚aA AX ,H$  ‚aA AX ,H$ [_ IENDB`׵(  ` X0Equation Equation.30,Microsoft Equation 3.00Equation Equation.30,Microsoft Equation 3.0 V/ 0|DTimes New Roman00X0||0DArialNew Roman00X0||0" DMonotype Sorts00X0||00DBook Antiquas00X0||0@DSymboltiquas00X0||0@  @@``  @n?" dd@  @@``   + )%&Y.C'%"# # #% $   33 $%&'# !"G3* (    +_b$;lOzr2$n| 5g~2$0p+|㐸.b$Z`z+oJ߻eb$uB &)YX$3S ~1? 33jl@>g4RdRd0j`ppp@ g40d0d0 appc  <4BdBd00? ,O =Q,!Coverage Metrics for Verification"n"}Serdar Tasiran, Kurt Keutzer Department of Electrical Engineering & Computer Sciences University of California, Berkeley D~ Z^Outline The need for coverage metrics Typical current use of coverage Survey of coverage metrics from Hardware verification (CAD) Protocol conformance testing Software testing L^lLaKThe Need for Coverage Metrics Good coverage metrics enable us to Guide test vector generation to exercise unchecked aspects of designs Quantify the comprehensiveness of verification achieved by a combination of techniques Model checking, simulation, symbolic simulation, ... Compare and correlate techniques, eliminate redundant effort Assess whether a certain degree of verification has been achieved Construct compact test suites. $l5lld$5(System Overview Typical Verification FlowtSpecify design and partition into functional blocks. For each block write interface and functional specifications Based on specs, generate functional test plan. List of scenarios, event combinations, etc. to be tested. Write code modeling surrounding environment:  stubs Generate test vectors (Biased) random generation Deterministic tests:  Diagnostics ,  Directed tests Simulate, check results Inspection Monitors (hand written or automatically generated) Comparison with reference model Get coverage information: Typically code or  FSM coverage Address uncovered aspects Write directed tests Change weights of random vector generator  How much is enough is decided by practical constraints.:KP^U?::-P^U?:Outline The need for coverage metrics Typical current use of coverage Survey of coverage metrics from Hardware verification (CAD) Protocol conformance testing Software testing t>l dlLAKDesired Qualities of MetricsFDesired qualities of a coverage metric Good correlation with design errors High coverage: Most possible bugs checked for Low coverage: Reasonable chance that there are errors Ease of use Ease of evaluation: little overhead Able to guide verification towards high coverage. Computationally feasible to achieve high coverage.b($f ($f Research ApproachY 1. Form of spec and implementation 2. Testing/verification goal 3. Error models 4. Coverage metrics 4.1 Does it correlate well with errors? Intuitively Experimentally Effectiveness: Likelihood of catching bugs given X % coverage. 4.2 Applicability How easy is it to achieve coverage? Likelihood of catching bugs vs. computation time 4.3 Relevance for hardware verification purposes 5. Comparison of metrics: theoretically and empirically X% C1 coverage vs. Y% C2 coverage. Computational cost, size of test set required, ...|#AEcAAEO(AEO@EO?@EOAEOU@EO1AEO8AEO#@EO3O(!((((%(0(U(.(6V8SummaryVery little knowledge of mentioned issues in CAD for hardware. Only intuitive or anecdotal justification of metrics Strictly formal/mathematical analysis of coverage metrics is unlikely to define one best or most comprehensive metric No such conclusion from existing work Existing results have the form  For testing scenario A, according to probabilistic measure P, metric C1 is more likely to detect errors than metric C2 At very least, many coverage metrics are incommensurable Directions for research Statistical/empirical data gathering Situation analogous to fault models in manufacture test Evaluate and compare metrics by arriving at good statistical data to support claims. Search for intuitively better coverage metrics.x?5vbFy9lv%d0d?5TF% -P$  Outlinew Survey existing coverage approaches from Hardware verification (CAD) Protocol conformance testing Software testing v+d@AAEE+/Rough TaxonomyFSM coverage: Cover state transition diagram of an FSM States Transitions Paths Netlist coverage: Exercise structural description of circuit Toggle each node Exercise registers, register to register paths Code coverage: Exercise HDL description of the circuit Line coverage Branch coverage Sub-expression coverage Toggle coverage Functional coverage: Exercise different modes of operation Exercise each required task Write and exercise monitors for specific temporal behavior7=v@7vFv}7=@7F}PCAD: Coverage Metrics CONTROL EVENT COVERAGE [Ho & Horowitz  96] FSM coverage for control variables controlling datapath Ge: Control event graph. Project control FSM onto variables in control-datapath interface No need to consider other control variables Assumptions: Design already partitioned into datapath and control Datapath does not hold any control state Automatically extract from Verilog (w/ user annotations) Comments in Verilog highlight important control state variables Transitive set-of-support: Capture logic that controls these variables Derive list of  independent variables, coverage tool will use thisdlp l`9*j5( (l `9@q>dD%Control Event Coverage: ApplicabilityCONTROL EVENT COVERAGE [Ho & Horowitz  96] Empirical data: In general, same test set gives less control event coverage than full state/event coverage Highlights important tests that are missed Full coverage analysis gives huge # of untested scenarios: Hard to use this data Using fewer variables and over-approximation, easier to incrementally construct a test scenarioZ<vw*jv Tag Coverage  Tag CoverageMechanism to extend standard code coverage metrics using observability requirements Errors in design are modeled as errors in assignments in HDL. IDEA: Tag a variable +D, -D: deviation from intended value Optimistic assumption: Deviation big enough to propagate in each case. Example: x > y + D Run a set of simulation vectors Tag one variable assignment at a time Use tag calculus. e.g. z= x + y If x has +D and y has -D z doesn t get tagged. Determine which ones propagate to the output Method confirms that tag is activated and propagated to an observable variable.\ x 2 >K  8 Tag CoverageFThere is full observability of internal nodes during simulation BUT this info may be incomprehensible even by designer Observability-based coverage gives more meaningful % numbers According to observability-based metric, user given test cases yield much better coverage than random Not necessarily the case if observability isn t considered Overhead for computing controllability not too much 1.5-4 simulation time Captures most errors that can be caught by structural metrics DISCLAIMER: Bugs do not always manifest themselves as an incorrect value of some HDL variable Errors of omission Wrong global assumptions Tag coverage is incomparable with almost all other metrics. @7bb;b4bb>b^,=@(3:4(> S,< 0-InChecker: Monitor described in HDL that observes Register leaks Register activity: Is each register Initialized? Loaded? Read? Register-to-register interactions: Are all feasible ones exercised? Datapath-control interface Counters: Overflow, underflow Record activity of each monitor. PAIR-STATE, PAIR-ARC COVERAGE View design as collection of interacting controllers Pair arc: a single-cycle interaction between a pair of controllers. Fundamental unit of controller-to-controller behavior. Extensively simulated designs often have < 10% pair-arc coverage Claim: Pair-arc coverage addresses corner cases Multiple unlikely events occurring simultaneously Timing errors Results on FLASH protocol processor support claim intuitively. 03}$#@A '3^ 5 *@ACAD: Recent Work > Coverage Estimation for Model Checking Hoskote, Kam, Ho, Zhao [DAC  99] Estimate the  completeness of a set of properties verified by MC Select  observed signal : Proposition or signal in circuit States covered by ACTL formula F: States where the value of the observed signal affects correctness of F Useful in uncovering missing properties if coverage < 100% 100% coverage must be achieved for several important variables, including ones that are related to the interactions of components. As with all model based metrics, can t point out missing functionality.  A Study in Coverage-Driven Test Generation Benjamin, Geist, et. al. [DAC  99] State and transition coverage of a hand-written abstract model. Manual abstraction: Error prone. Difficult to map uncovered portions to refined model.JbPbUD) I-"UDP).CAD: Recent WorkXEnhancing Simulation with BDDs and ATPG Ganai, Aziz, Kuehlmann [DAC  99] Test generation for coverage of control blocks in HDL code. Guidepost Coverage Yang, Dill [DAC  98] Set intermediate goals for reaching designated state set Bias search to maximize achievement of intermediate goals Useful: captures intuition about how to reach a state inside big state-space Requires designer effort. User-defined Coverage Grinwald, Harel, Orgad, Ur, Ziv [DAC  98] (IBM) User defines  coverage tasks using simple language: First-order temporal logic + arithmetic operators Snapshot tasks: Condition on events in one clock cycle Temporal tasks: Refers to events over different clock cycles  I=(G52u( =1 2',  bCommercial Coverage ToolsYSureFire (SureCov), Design Acceleration Inc. (Coverscan), SummitDesign, TransEDA (HDLCover, VeriCov), interHDL (CoverIt), Veritools, Blocks, arcs (branches), expressions, FSM states and transitions, sequences, pair FSM coverage Covermeter Statement, block, branch, conditions coverage Register and net toggle coverage FSM coverage Data transfer coverage (register transfer and buses) ?? Invariant coverage/ assertion checking ?? SureFire (SureSolve) Functional verification suite and automatic testbench generation. Exercises 90 to 100% of all reachable HDL constructs !! Verisity + CBL (!), VERA (Synopsys) Functional coverage: User defined monitors. Test generation can make use of coverage information.j`_ vz'vb  _ \a L$ b<d0Outlinex Survey existing coverage approaches from Hardware verification (CAD) Protocol conformance testing Software testing +@Ad@AAEE+!Protocol Conformance/FSM Testing "Y"SPEC: Single Deterministic (Extended) FSM Written in ESTELLE, LOTOS, SDL IMPL: Single Deterministic (Extended) FSM running inside bigger system EFSM: FSM describes control. Extra state variables: Data structures Context variables. Mostly small control part: <100 states. Bigger data part. 50-100 input and output variables VERIFICATION GOAL: IMPL is equivalent to or contained in SPEC Not only I/O behavior: States and transitions must match State of IMPL not visible Black box testing of IMPL vs. SPEC +@EY@EYO@EY0@EY#@EY@EYEAEdxAEmY#dY@EY& C0#C("B Protocols: Datapath Coverage Definition-Use Coverage Determine all definition- use pairs May be an exponential # in the # of branches For each definition-use path Get to its initial state Determine if the d-u path is traversable Constraint solving (SAT like approach) May need to traverse loops several times to make path traversable Control + Datapath: Repeat same control test with different data values All control states/branches + all definitions-uses Cover all d-u paths first Cover remaining transitions AEY$@EY/@EY@E'B@EYi@EY@Emg@EY6@EY@EY%/Big6+Protocols: A Metric for Execution Sequences,Y+L[Zhu, Vuong  97] Infinite behavior space, want to cover w/ finite tests Execution sequences: a1, a2 , & , an . b1, b2 , & , bn . Define distance between sequences d(a1,b1) + 1/2 d(a2,b2) + 1/4 d(a3,b3) + & 0 d(ai,bi) 1 Obtain a metric space Cover space with finitely many spheres of radius e. Rationale: Differences later on in the sequence don t matter as much Can pick test sets of decreasing e. Problem: If metric says two sequences are close, are they really close in terms of bug-catching probability? Generalize to data parameters: (a1(v1,& ,vn), r1), (a2(v1,& ,vm), r2), & Distance metric depends on v1,& ,vm as well No results on applicability.NAEY@EY,@EY@EY@EY4@EOE@EY@EY_@Ed+@EY@EYM        #          G ;!            z+ Protocols: Comparison of Metrics!Y!C1 subsumes C2 iff " programs P, " test sets T, T gives 100% C1 cov. T gives 100% C2 cov. C1 subsumes C2 A test set with 100% C1 is more likely to uncover bugs than C2. Elaborate metrics more effective at X% coverage because Large test set needed for X% coverage Branch coverage vs. random testing: Not much improvement for comparable test set size. All definitions-uses coverage: Requires a lot of computation for high coverage. In one study 40 d-u tests is as effective as 250 random tests. D-U tests require computation. Random testing not as good as path testing But a good enough cost-effective alternativeD-@EY&@EY%@EY3@EY @EY?@EYR@EY+@EZ.@EZ(  ' (  (I;%#3?R+.Outliney Survey existing coverage approaches from Protocol conformance testing Hardware verification (CAD) Software testing +;@AdAEE+;$SW Covg. Metrics (Adequacy Criteria)%Y%*Classification 1: Program-based Spec-based Combined Interface-based: Inputs and outputs adhere to required format Random (statistical) testing Probability distribution over input space Classification 2: Structural metrics Exercise program text Fault-based metrics Inject local faults Error-based metrics Exercise error-prone points in program Unit testing only. Some recent work in  integration testing @EYa@EY@EY*@EY@E@EY@EY@EY@EY@EY'@EY=@Ew.*'&$SW: Program-based Structural Testing CONTROL-FLOW-BASED METRICS Basics: Statement coverage Branch coverage Decision (multiple condition) coverage Path coverage. 1, 2 and 3 usually too weak. Miss errors. Also undecidable: Code may be unreachable 4 ideal but impractical: Too many paths Choose representative subset of paths Simple/elementary paths Length-n paths Level-i paths Check all elementary paths At next level, check unexercised elementary subpaths and cycles Only check a linearly independent set of paths There are = e - n + # of SCCs of them (Cyclomatic number)dd'dd+*Ub6[0: Y(( *(,(&6[0 (($SW: Program-based Structural TestingDATA-FLOW-BASED METRICS GOAL: Don t fold in data variables into state space Find meaningful and efficient way to exercise them Definition occurrence x := 3 Use occurrence (Global use) Computational use z := x + 1 Predicate use if (x >2) then ... All-definitions: Each definition of variable exercised by some path. All-uses: Each (feasible) use of a variable is covered by a path Computational uses Predicate uses All c-uses, some p-uses All p-uses, some c-uses ... There may be many paths through which definition reaches use "bP}V>`,- 6 4 6V>g%SW: Program-based Structural Testing DATA-FLOW-BASED METRICS (cont d) All definition-use pairs: " definitions of x " (feasible) paths q via which def n reaches a use of x, $ path p such that q is a sub-path of p. Interactions between different variables must be exercised [Ntafos] k def-ref interactions: [d1(x1),u1(x1), d2(x2),u2(x2), d3(x3), & , dk(xk),uk(xk)] di: def of variable xi ui: use of variable xi ui and di+1 are at the same node di reaches ui xi s and nodes nj need not be distinct Interaction path for k-def-ref p = Required k-tuples criterion For all j-def-ref interactions L with 1 j k, there is an interaction path for all feasible Ln!iNDQ]Ma%'  .(        (    ( ((( (( ((( (( (( (( ( ( (( ( ( (+( ((((((((0(>;qG~!SW: Fault-Based Adequacy CriteriaStructural testing Fault-based testing: Adequacy of test set = Ability to detect faults Methods: Error seeding Mutation testing Program mutation testing SPEC-mutation testing Error-based testing4dm9@EY/@E)8.SW Metrics: Mutation Coverage (rProcedure for program P Create set of alternative programs: MUTANTS Perturb program locally by mutation operators Construct test set T For each mutant M, run tests from T Either P and M differ on a test Mutant dies Or T is exhausted Mutant lives Live mutants provide valuable info. Mutant lives because Mutant is equivalent to program Only a small fraction should be this way Test data inadequate Large proportion of mutants live, Test data does not convince us that P is correct Live mutants point to  un-exercised aspects of program Mutation adequacy =,.9T9v *Wd83%(.(((((( (( 9 *$(%(CSW: Generating MutantsMutation operator Replace one syntactic structure with another Designed based on previous design experience Statement Analysis: Make sure every line and branch is necessary Replace statement with CONTINUE, TRAP Replace logicals and relationals with TRUE or FALSE Replace DO with FOR, etc. Predicate Analysis Exercise predicate boundaries Alter limit sub-expressions by small amounts: e.g. x < 5 Insert absolute value operators into predicate sub-expressions Alter relational operators Domain Analysis Change constants and sub-expressions by small amounts Insert absolute value operators (if syntactically correct) Coincidental Correctness Analysis Change data references and operators to alternatives (wherever syntactically correct)ZAtq"WZA'3[q"W Mutation Analysis: ApplicabilityZPROS Easy automation Applying mutation operators Running mutants Interactive test environment. If mutant doesn t fail while original fails, easy to examine mutation and determine if there is an error. Other testing methods special cases of mutant testing Example: Statement and branch coverage CONS Expensive (both in time and space) n-line program = O(n2) mutants (Empirical result) Human cost of examining live mutants Empirically ~10% of all mutants Must decide if they re equivalent to original program If not, must create new test case to kill mutant,j6'l#2&,j6#((((& (E0SW Coverage Metrics: Comparison Only statistical measures for  better bug detection ability No formal relationship that says  metric X is better at detecting bugs than metric Y . Little experimental data to compare effectiveness. A few data points: Error exposing ability good only at high coverage levels 100% branch coverage not effective enough for protocols For general software 70% is considered excellent. For comparable test size Sophisticated metric not more effective Requires extra computation Difficult to achieve high coverage for elaborate metrics May require large test set Sophisticated metrics better at finding difficult bugs=@EYAEY9@EY82@EY@EmC@EY9@Em@EY7@Em*3)383s2C97SummaryTVery little knowledge of mentioned issues in the CAD world. Strictly formal/mathematical analysis of coverage metrics is unlikely to define one best or most comprehensive metric No such conclusion from existing work Existing results have the form  For testing scenario A, according to probabilistic measure P, metric C1 is more likely to detect errors than metric C2 At very least, many coverage metrics are incommensurable Directions for future research Statistical/empirical data gathering Situation analogous to fault models in manufacture test Evaluate and compare metrics by arriving at good statistical data to support claims. Search for intuitively better coverage metrics.^<vbFy9lv%d0dF% -W$  /T9Z ` ̙33` 3` 3333f` 999MMM` f` f3` 3>?" dX.?lD(X.us." dD  [ n?" dd@   @@``PR   . D [` p>  >        " (     `gֳgֳ ?``  =*   `gֳgֳ ?   _!Tasiran and Keutzer, UC Berkeley""!   `gֳgֳ ?`   ?*8 |  |   B,CrDEF1?++Yq @|   BWCDEFt1?++V @|   B,CrDEF1?+Y+q+ @   BWCDEFo 1?V++V @|~   Ni @1?   Z D[D[ ?w   T Click to edit Master title style! !#   T!D[D[ ?j  ;Body Text Second Level Third Level Fourth Level Fifth Level     <H  0޽h? ? nCסnC3e stari95.ppt 0 h`@(    Tll7ll7 ?#   ]* H$$HHll  T$ll7ll7 ? #  _* H$$HHllj  s *1 ?m  :  Tll7ll7 ? I>n  RClick to edit Master text styles Second level Third level Fourth level Fifth level!     S  Zll7ll7 ?#   ]* H$$HHll  ZDll7ll7 ? #  _* H$$HHllH  0l(? ? ̙33 0 D(      TDll7ll7 ?#   ]* H$$HHll  Tll7ll7 ? #  _* H$$HHll  Zll7ll7 ?#   ]* H$$HHll  Zdll7ll7 ? #  _* H$$HHllH  0l(? ? ̙33t $P(     fLD[D[ ? Y)     fMD[D[ ?  PP   H  0޽h ? nCסnC3e  `(  l  C S w    l  C N j  H  0޽h ? nCסnC3e  p$( w r  S $T w    r  S S ,  H  0޽h ? nCסnC3ew   '   ( w "  TdSo?< > r  S R w      ZDRgֳgֳ>?! cSimulation driver (vectors),Z  ZQgֳgֳ>?T [Simulation monitors,Z  ZPgֳgֳ>?\   ISimulation engineZ2  TDO>?' P ^Simulation model (HDL),Z|B  TD>?e ;|B   TD>?}|B  @ TD>?xxL    ZVgֳgֳ)?L  i X Diagnosis of Unverified Portions!Z!   ZVgֳgֳ>?L   ICoverage AnalysisZ|B  @ TD>?    ZdVgֳgֳ>?L =  IVector GenerationZ|B @ TD>? ! |B  TD>?L   ZDU(o?L  ROUR FOCUS NOW((H  0޽h ? nCסnC3e  (  l  C U w    l  C $N j  H  0޽h ? nCסnC3e  $(  r  S P w    r  S 4Q j  H  0޽h ? nCסnC3e  $( w r  S Q w    r  S Q j  H  0޽h ? nCסnC3e  U( w   # lRD[D[ ? w      # lSD[D[ ? r     TSD[D[ ?D^^ AFor each domain@EO  T4TD[D[ ?XD^~ <@EO H  0޽h ? nCסnC3e  T(  Tl T C T w    l T C TU j  H T 0޽h ? nCסnC3e  $( w r  S V w    r  S tV ,  H  0޽h ? nCסnC3e  ( 3t  l  C 4W w    l  C W j  H  0޽h ? nCסnC3e  `.(  ` ` # lTXD[D[ ? w     ` # lXD[D[ ? j  f ` s *A ?? H ` 0޽h ? nCסnC3e  0(d(  d d # l4ZD[D[ ? w     d # lZD[D[ ? j  H d 0޽h ? nCסnC3e  UM h(  hr h S \ w     h s }2.p%A`C:\WINNT\Profiles\serdar.000\Desktop\img007.gif  h C xA`C:\WINNT\Profiles\serdar.000\Desktop\img009.gifE  h Tt\1?%J 0 Observability-based Coverage Metric [Fallah, Devadas, Keutzer  96]4E 2&P  h  `1? h  !Example courtesy of Farzan Fallah&"!H h 0޽h ? nCסnC3e  0l@( 1 l l # lD[D[ ? w     l # lDD[D[ ? j   l oPA`C:\WINNT\Profiles\serdar.000\Desktop\img017.gif E|N  l q]YK~ A`C:\WINNT\Profiles\serdar.000\Desktop\img017.gifG I|G H l 0޽h ? nCסnC3e  0(@p(  p p # lD[D[ ? w     p # lD[D[ ? j  H p 0޽h ? nCסnC3e  P0(  r  S Ĕ w    ~  s *$(  j  H  0޽h ? nCסnC3e  `$(  r  S  w    r  S  j  H  0޽h ? nCסnC3e  pt$( @ tr t S d w    r t S ė j  H t 0޽h ? nCסnC3e  0(x(  x x # lD[D[ ? w     x # lD[D[ ? j  H x 0޽h ? nCסnC3e  $( w r  S Ě w    r  S $ ,  H  0޽h ? nCסnC3e  x( w   # lD[D[ ? w      # lDD[D[ ? j  HF _ `     `/ ~"  N1?_ ` ~2  N1?i.M~2  N1?B  ZDo?9B   ZDo?%PB  B ZDo?oB   ZDo?P;    fa1?z Q i = 1 / o = 0(      f:ĝ1?V M x := wait(  T a < # 22  To?a .   `1?b < Zs3<T a < # npB 2  To?a .   `1?b < Zs4<T a < # o2  To?a .   `d1?b < Zs2<T a < #  2  To?a .   `1?b < Zs1<2  To? 5     `$1? < ]sinit<   `1?I>9 e!i = 0, x = wait / o = 1 x := done("! H  0޽h ? nCסnC3e  wo(  r  S  w    r  S  j  |"  T1?  |2  T1?wx|2  T1?B   `D(o?B   `D(o?YB  @  `D(o?neB    `Do?OW"   # la1?Ug Q i = 1 / o = 0(     # l:1? M x := wait(  R a .  3 |2  Zo?a .   f1?b  ` Zs3<R a . 3 dx 2  Zo?a .   fD1?b  ` Zs4<R a . 3 k~2  Zo?a .   f1?a a Zs2<R a . 3 G Z2  Zo?a .   f1?b  a Zs1<2  Zo? K-    fd1?#  ]sinit<   f$1?. e!i = 0, x = wait / o = 1 x := done("!    `1?, N Definition(  (B  ZD(1?   `D1?~3  GUse( (B  ZD(1? xwH  0޽h ? nCסnC3e  0((    # ldD[D[ ? w      # lD[D[ ? j  H  0޽h ? nCסnC3e  D( dhĊ   # l~D[D[ ? w      # l$D[D[ ? j  |B @ TD(Ԕ?gDH  0޽h ? nCסnC3e  P$(  Pr P S D w    r P S  ,  H P 0޽h ? nCסnC3e  0((( w ( ( # ldD[D[ ? w     ( # lāD[D[ ? j  H ( 0޽h ? nCסnC3e"    8b(  8 8 # lD[D[ ? w     8 # lDD[D[ ? j  F C  8 R B 8 ZDo?CB 8B ZDo?Y B 8 ZDo?K B 8 ZDo?c  B  8B ZDo?K    8  BCDEF"jJ?W@`-ZJB  8 ZDo?L  8  `1?  =Level 2  8  `1?G ILevel 1&H 8 0޽h ? nCסnC3e  0(<(  < < # lDD[D[ ? w     < # lD[D[ ? j  H < 0޽h ? nCסnC3e\,   ,, FF@+(  @v" @ N1?K v" @ N1?R Sv" @ N1?&J K  @ # lD[D[ ? w     @ # ldD[D[ ? j  F - @  o2 @ ZćjJ?n Fx:=3&2  @ Z$jJ?/ Gx > 2&B  @B ZD1?4B  @B ZD1?B  @ ZD1?B  @B ZD1?hg4B @ ZD1?a4B @ ZD(1?,B @B ZD(1?'B @B ZD(1?55 @  `1?\ Rq14  @  `1?\ < ^q2@ @  `D1?M - Zq3<((B @ ZD1?KJB @ ZD1?pB @ ZD1?hwB @B ZD1?YB @ ZD1?aoB @B ZD1?vZ~2 @ N1?K~2 @ N1? ~2 @ N1?~2 @ N1?~2 @ N1?p~2  @ N1?(Z[2 !@ T1?L5h B ~2 "@ N((1?j~2 #@ N((1?7 WF a & sJ  $@ & a sJ N R N d6  %@ a * s N    &@ R N S 5 ~" '@ NjJ? u  (@  `1?   Rn14 N    )@  N 5 ~" *@ NjJ? u  +@  `d1?   Rn24 N    ,@ 7N 85 ~" -@ NjJ? u  .@  `T 1?   Rn34 N    /@ cO d6 ~" 0@ NjJ? u  1@  ` 1?   Rnk4 TN < - H  2@ < - H d 3@  4BCDEXF(jJ?qM*3j 7[$:PHXhxmc@       < @ = ~2 4@ N1?v - \ ~2 5@ N1?  ~2 6@ N1?6  e H TN < - H  7@ . SI d 8@  4BCDEXF(jJ?qM*3j 7[$:PHXhxmc@       < @ = ~2 9@ N1?v - \ ~2 :@ N1?  ~2 ;@ N1?6  e H TN < - H  <@ '' B d =@  4BCDEXF(jJ?qM*3j 7[$:PHXhxmc@       < @ = ~2 >@ N1?v - \ ~2 ?@ N1?  ~2 @@ N1?6  e H TN < - H  A@ / J d B@  4BCDEXF(jJ?qM*3j 7[$:PHXhxmc@       < @ = ~2 C@ N1?v - \ ~2 D@ N1?  ~2 E@ N1?6  e H  F@  ` 1?&   C...$H @ 0޽h ? nCסnC3e  0(04( w 4 4 # l D[D[ ? w     4 # l D[D[ ? j  H 4 0޽h ? nCסnC3e  @.(    # lTD[D[ ? w      # lD[D[ ? j  f  s *A ?? R H  0޽h ? nCסnC3e  0(P(    # lD[D[ ? w      # l4D[D[ ? j  H  0޽h ? nCסnC3e  0(`(    # lD[D[ ? w      # lD[D[ ? j  H  0޽h ? nCסnC3e  0(p(    # lD[D[ ? w      # lD[D[ ? j  H  0޽h ? nCסnC3e! !! 22D'!( w D D Zgֳgֳo? y\ { Fstatement coverage D Zgֳgֳo?d  J = all-paths  D ZTgֳgֳo?F Eall level-i paths D Zgֳgֳo? E O  Cbranch coverage D Zgֳgֳo? \x]  Gcyclomatic adequacypB D@ Ho? vpB D@ Ho?@ " pB  D Ho? pB  D Ho? pB  D@ Ho? pB  D Ho?[  pB  D@ Ho?N  pB D@ Ho?E  D Zgֳgֳo?x DOrdered context D ZTgֳgֳo?  A all d-u paths D Zgֳgֳo?N  Erequired k-tuples D Zgֳgֳo?9 Cstrong mutation D Zgֳgֳo?~1 H(un-ordered) context D Zgֳgֳo?# Jall-p-uses some-c-uses D ZDgֳgֳo?w P,]  > all-p-uses  D Zgֳgֳo?-  = all- uses  D Zgֳgֳo?  o  Jall- def ns  D Zgֳgֳo?^ w Brequired pairs D ZDgֳgֳo?{   Lall-c- uses some-p- uses D Zgֳgֳo?6 A firm mutation D Zgֳgֳo? /]  A weak mutationpB D@ Ho?pB D Ho?) pB D@ Ho?6~ pB D Ho?  1pB  D Ho? t gpB !D Ho?wxpB "D@ Ho?^ R pB #D Ho?pB $D@ Ho?{ pB %D@ Ho?` 3  pB &D Ho?\  pB 'D@ Ho? ` pB (D@ Ho? V  L | )D# q v  *D  B,CrDEF1?++Yq @| +D  BWCDEFt1?++V @| ,D  B,CrDEF1?+Y+q+ @ -D  BWCDEFo 1?V++V @|~ .D Ni @1?B /D  `$1?  R Subsumes relation between criteria (5)D*%(( 0D Zgֳgֳo? u '  ? all-c- uses pB 1D@ Ho?ahz pB 2D Ho? H D 0޽h ? nCסnC3e  $(  r  S  w    r  S  j  H  0޽h ? nCסnC3eV, 0 (     T1 ?m     # lll7ll7 ? I>n    H  0l( ? ̙33b 0 "|(  | |  Z1 ?m    | 3 r4dl4dl4 ? I>n    H | 0l( ? ̙33b 0 "(     Z1 ?m     3 rdl4dl4 ? I>n    H  0l( ? ̙33xMOQpۥmbQ‰1JQ U]Cw)ݶ <=O` &Ɠ^LG{rm07_og힝.FM< y]o4q ~ v;4Y1 9 p:z|%K4]2wOAr#)Waܶ|6R4W"1,a7N{?@J'ڿ](Ňl?:_w fN.$x1|v3}[`' Kǿ)rP B;W,;(Q49S-Fģb<W*M V]UI%JOʈ]LhBQ#bye$MimgzJd`/K#x=нC{0儷jK8vTw}Z8amIu EM#̔ nλU{&(8>uCuڥ6`Z3V,B9t" Erן酂[,{gh% ^P$M.n'$7 ku-> tY5RB˵ /D\z~ĜXC$=} W%n `u䪋 #ڈ2b|#0w!VXke,RRTrxep; ϫJ[cmュpفjxMoWp[$$$BHD *Eĕ*Q 6sbw`GpBJTV=p*Zvgޛ]G2Ѿy3of{3k77XjB6'5$ӭvuY B\t5/_L #&E@Jݿb'e;? ٨|UP]L R$v Dk0~Lo@TiNi|ے W^Av/= 3]'_W-n%󟶆j@b/!APZ3t#PL iLAc1۩O[ܧ8R _ɅnX=gnJ>^1B:dB9&m; d\l,(G4"WF-UָwdGDfbb{6sQN`QcX&wH&GpԜrwOg}iF`0%+yj `(y@:%,#PT9~`.x2g5:Ǿk*c}Jݼ=H{ mⰮD9'[Ill* [L!B3~5&-Q$rg骊V ZNEzЭj/=opg3$&b "{s@樵UЧ:Szjx,_ oN ^~<o9-W [ȥȥEĐ>;3L7Ubܬ<z r`LW#:B6t&'\:iQ)=8 .*xxF[x͗f?K &VDViq~}$){`>QXᒰxr0A ߵ9ڨy%\ 9 @qzg}w`K=w?HK_;%PX 8Y@pYp8 $ 5S8E, Oh+'0L px  0 < H T`h3Computing Delay with Coupling Using Timed Automatai CAD GroupDeC:\presentations\stari95.ppt Us CAD Groupat708Microsoft PowerPointi95@Fi@p ޾@:ө@hR GoM  f[& &&#TNPP0D & TNPP &&TNPP    --- !---&>zX&--xG-- "!w*wgw " - "Arial 3!w*wgw 3 - nC.92 !Coverage Metrics for Verification       .--99-- nCBook Antiquaw*wgw # - .42 *Serdar Tasiran, Kurt Keutzer       .Book Antiquaw*wgw 4 - .[2 8Department of Electrical Engineering & Computer Sciences              . .:2 4"University of California, Berkeley        .--"Systemwf3  -&TNPP &՜.+,D՜.+,@     wOn-screen Show UC Berkeley@% ,Times New RomanArialMonotype Sorts Book AntiquaSymbol stari95.pptMicrosoft Equation 3.0"Coverage Metrics for VerificationOutlineThe Need for Coverage MetricsSystem OverviewTypical Verification FlowOutlineDesired Qualities of MetricsResearch ApproachSummaryOutlineRough TaxonomyCAD: Coverage Metrics &Control Event Coverage: Applicability Tag Coverage Tag Coverage Tag Coverage0-InCAD: Recent Work CAD: Recent WorkCommercial Coverage ToolsOutline"Protocol Conformance/FSM Testing Protocols: Datapath Coverage,Protocols: A Metric for Execution Sequences!Protocols: Comparison of MetricsOutline%SW Covg. Metrics (Adequacy Criteria)%SW: Program-based Structural Testing%SW: Program-based Structural Testing&SW: Program-based Structural Testing "SW: Fault-Based Adequacy CriteriaSW Metrics: Mutation Coverage SW: Generating Mutants!Mutation Analysis: Applicability SW Coverage Metrics: ComparisonNo Slide TitleSummary  Fonts UsedDesign TemplateEmbedded OLE Servers Slide Titles%xu}     !)19AIQY _PID_GUID _PID_HLINKS TemplateType GraphicType Compression ScreenSize ScreenUsage MailAddress HomePage Other DownloadOriginal DownloadIEButtonUseBrowserColor BackColor TextColor LinkColor VisitedColorTransparentButton ButtonType ShowNotes NavBtnPos OutputDirAN{6D038AB5-AF4E-11D2-A051-00A0C967EE45}A*http://www.surefirev.com/SCDataSheet.htmldserdar@ic.eecs.berkeley.edu)http://www-cad.eecs.berkeley.edu/~serdartmlttp  f3 Z:\public_html\cbl.!_ CAD Group  !"$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~     Root Entry .dO) Kurt Keutzer Pictures   .Book Antiquaw*wgwE-Current User.[2 8ectrical Engineering & Computer Scie SummaryInformation   (   .PowerPoint Document2 4"University of (eley    #& DocumentSummaryInformation-8"Systemwf3  -&TNPP & f[