1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
|
\input texinfo @c -*- texinfo -*-
@settitle FATE Automated Testing Environment
@titlepage
@center @titlefont{FATE Automated Testing Environment}
@end titlepage
@top
@contents
@chapter Introduction
FATE provides a regression testsuite embedded within the Libav build system.
It can be run locally and optionally configured to send reports to a web
aggregator and viewer @url{http://fate.libav.org}.
It is advised to run FATE before submitting patches to the current codebase
and provide new tests when submitting patches to add additional features.
@chapter Running FATE
@section Samples and References
In order to run, FATE needs a large amount of data (samples and references)
that is provided separately from the actual source distribution.
To inform the build system about the testsuite location, pass
@option{--samples=<path to the samples>} to @command{configure} or set the
@var{SAMPLES} Make variable or the @var{LIBAV_SAMPLES} environment variable
to a suitable value.
To use a custom wrapper to run the test, pass @option{--target-exec} to
@command{configure} or set the @var{TARGET_EXEC} Make variable.
The dataset is available through @command{rsync}, is possible to fetch
the current sample using the straight rsync command or through a specific
@ref{Makefile target}.
@example
# rsync -aL rsync://fate-suite.libav.org/fate-suite/ fate-suite
@end example
@example
# make fate-rsync SAMPLES=fate-suite
@end example
@chapter Manual Run
FATE regression test can be run through @command{make}.
Specific Makefile targets and Makefile variables are available:
@anchor{Makefile target}
@section FATE Makefile targets
@table @option
@item fate-list
List all fate/regression test targets.
@item fate-rsync
Shortcut to download the fate test samples to the specified testsuite location.
@item fate
Run the FATE test suite (requires the fate-suite dataset).
@end table
@section FATE Makefile variables
@table @option
@item V
Verbosity level, can be set to 0, 1 or 2.
@table @option
@item 0
show just the test arguments
@item 1
show just the command used in the test
@item 2
show everything
@end table
@item SAMPLES
Specify or override the path to the FATE samples at make time, it has a
meaning only while running the regression tests.
@item THREADS
Specify how many threads to use while running regression tests, it is
quite useful to detect thread-related regressions.
@item THREAD_TYPE
Specify which threading strategy test, either @var{slice} or @var{frame},
by default @var{slice+frame}
@item CPUFLAGS
Specify a mask to be applied to autodetected CPU flags.
@item TARGET_EXEC
Specify or override the wrapper used to run the tests.
@item GEN
Set to @var{1} to generate the missing or mismatched references.
@end table
@example
make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate
@end example
@chapter Automated Tests
In order to automatically testing specific configurations, e.g. multiple
compilers, @command{tests/fate.sh} is provided.
This shell script builds Libav, runs the regression tests and prepares
a report that can be sent to @url{http://fate.libav.org/} or directly
examined locally.
@section Testing Profiles
The configuration file passed to @command{fate.sh} is shell scripts as well.
It must provide at least a @var{slot} identifier, the @var{repo} from
which fetch the sources, the @var{samples} directory, a @var{workdir} with
enough space to build and run all the tests.
Optional submit command @var{fate_recv} and a @var{comment} to describe
the testing profile are available.
Additional optional parameter to tune the Libav building and reporting process
can be passed.
@example
slot= # some unique identifier
repo=git://git.libav.org/libav.git # the source repository
#branch=release/10 # the branch to test
samples=/path/to/fate/samples
workdir= # directory in which to do all the work
fate_recv="ssh -T fate@@fate.libav.org" # command to submit report
comment= # optional description
build_only= # set to "yes" for a compile-only instance that skips tests
# the following are optional and map to configure options
arch=
cpu=
cross_prefix=
as=
cc=
ld=
target_os=
sysroot=
target_exec=
target_path=
target_samples=
extra_cflags=
extra_ldflags=
extra_libs=
extra_conf= # extra configure options not covered above
#make= # name of GNU make if not 'make'
makeopts= # extra options passed to 'make'
#tar= # command to create a tar archive from its arguments on
# stdout, defaults to 'tar c'
@end example
@section Special Instances
The @var{TARGET_EXEC} option provides a way to run FATE wrapped in
@command{valgrind}, @command{qemu-user} or @command{wine} or on remote targets
through @command{ssh}.
@section Submitting Reports
In order to send reports you need to create an @command{ssh} key and send it
to @email{root@@libav.org}.
|
