OpenCores
URL https://opencores.org/ocsvn/radiohdl/radiohdl/trunk

Subversion Repositories radiohdl

[/] [radiohdl/] [trunk/] [doc/] [radiohdl_programmer_guide.md] - Blame information for rev 7

Details | Compare with Previous | View Log

Line No. Rev Author Line
1 7 danv
# RadioHDL Gear Programmer Guide
2
 
3
### *Speed up HDL development*
4
 
5
---
6
#### Document history:
7
|Revision|Date|Author|Affiliation|Modification|
8
|:---|:---|:---|:---|:---|
9
| 0.9|18 sep 2018|R. Overeem |ASTRON|HDL programmers manual, created after refactoring the Python3 and bash code.|
10
| 1.0|27 jan 2020|E. Kooistra|ASTRON|Converted HDL programmers manual docx into this md file|
11
 
12
---
13
#### Contents:
14
1 Introduction
15
1.1 Purpose
16
1.2 Design principles
17
1.2.1 Rules for scripts and configuration files
18
1.2.2 Rules for keys and values
19
2 Handling configuration files
20
2.1 Base classes
21
2.1.1 ConfigFile
22
2.1.2 ConfigTree
23
2.2 Related classes
24
2.3 Unit tests
25
2.4 Utility modify_configfiles
26
2.5 Relation with old CommonDictFile
27
3 Initialisation of the environment
28
3.1 Usage of configuration file settings in a shell
29
3.2 set_quartus.sh and set_modelsim.sh
30
4 Programming principles
31
4.1 Shell scripts
32
4.1.1 Shell options
33
4.1.2 generic.sh
34
4.1.3 Parsing arguments
35
4.2 Python
36
4.2.1 Importing packages
37
4.2.2 Limiting export
38
4.2.3 Parsing arguments
39
 
40
---
41
#### References:
42
[1] radiohdl_user_guide.md
43
[2] radiohdl_hdl_buildset_key_descriptions.md
44
[3] radiohdl_hdl_buildset_uniboard1.md
45
[4] radiohdl_hdl_tool_modelsim.md
46
[5] radiohdl_hdl_tool_quartus.md
47
[6] radiohdl_hdl_library_key_descriptions.md
48
 
49
---
50
## 1 Introduction
51
### 1.1 Purpose
52
This document describes the inside gear of the RadioHDL package [1].
53
 
54
### 1.2 Design principles
55
Originally the RadioHDL gear consisted of Python2 and bash. The conversion to Python3 was taken as an opportunity to refactor the RadioHDL code. These design principles also apply to future updates and extensions of the RadioHDL package, see also section 4.
56
 
57
#### 1.2.1 Rules for scripts and configuration files
58
* Keep the shell environment as clean as possible
59
Therefore RadioHDL uses **init_radiohdl.sh**, to (manually) start RadioHDL in a terminal.
60
 
61
* Store all 'user tuneable' variables in configuration files instead of in scripts
62
This avoids too much tool knowledge in scripts and eases the maintenance of the scripts.
63
 
64
* The scripts must clearly report if they fail
65
For example the Python scripts give a hint of what causes the failure. Bash scripts report if they fail, instead of finishing silently.
66
 
67
* Avoid classes that can do too much
68
Instead Use classes that 'embed/guard' some data(structure) and provide a few functions for manipulating this data in a controlled way. This normally means that a class is very good in only one or two things. The maintenance of multiple dedicated classes is much easier, since it is very clear what each class does. This rule is applied in section 2.
69
 
70
#### 1.2.2 Rules for keys and values
71
* Source or target oriented keys
72
Whether a key is source oriented or target oriented depends on whether its files are used for one or more targets. In general if a file is used for more targets then source oriented is preferred to avoid having to list the file name twice. If a file is used only for one target then target oriented is preferred to be more clear about the purpose of the key. For example the quartus_* keys in the hdllib.cfg [5], [6] are now source oriented. Instead it may be better to redefine them as target oriented. E.g. a 'quartus_create_qsf' key that defines to create a qsf file using the information listed in the values.
73
 
74
* Avoid hidden behaviour of keys
75
The 'synth_top_level_entity' key in the hdllib.cfg [6] enforces the creation of a qpf and qsf. This kind of hidden behaviour is not so nice.  Instead it is more clear to have an explicit 'quartus_create_qpf' and 'quartus_create_qsf' key to define this.
76
 
77
* Support new key names
78
Unknown key names are ignored, such that multiple tool scripts can use the same cfg file. Each tool only handles the keys that it knowns. Hence a key name will only cause an action if it is known by a tool script. Therefore key names must be predefined and can be described in a tool configuration file specification document.
79
 
80
* Support more value definitions for keys
81
Currently keys can have one value, a list of values or a list of pairs [1]. More value definitions could be added to the RadioHDL configuration file schema, e.g. a list of value tuples, whereby a tuple can contain more than a pair.
82
 
83
* Executing key values
84
An issue can be that it can be dangerous to blindly execute a key value, because it is user defined and could be a script that contains e.g. 'rm -rf ~/*'.  E.g. 'quartus_tcl_files' sources a tcl script, it is left to the user to ensure that this tcl script is a legal tcl script. For shell commands blind execution of commands can be prevented by defining a dedicated key per command, such that it is impossible to execute a key value, so instead of e.g. 'ls *' the key value then become '*' and the key name itself invokes 'ls'.
85
 
86
* Section headers
87
Support for new tools may be added by using a dedicated [section header] in the configuration file.
88
 
89
---
90
## 2 Handling configuration files
91
The handling of the configuration files involves e.g.:
92
 
93
- represent the content of one configuration file
94
- represent the content of a whole hierarchical tree of configuration files
95
- a bulk modification tool to modify keys and/or values
96
 
97
This diverse functionality has resulted in eight  classes and one interactive command line tool. Figure 1 shows the class diagram.
98
 
99
![Figure 1](./configuration_file_classes.jpg "configuration_file_classes.jpg")
100
Figure 1: Class diagram of the configuration file classes.
101
 
102
### 2.1 Base clases
103
The main functionality located in two base classes *ConfigFile* and *ConfigTree*.
104
 
105
#### 2.1.1 ConfigFile
106
The ConfigFile base class can read in one configuration file and store that content to an OrderedDict inside the class. This is done during the construction of the class. If the read in fails a ConfigFileException is thrown.
107
 
108
* Arguments:
109
  - filename : full filename including the absolute path of the file
110
  - sections : optional argument to limit the sections that are read in.
111
  - required_keys : optional list of key names that must exist in the configuration file.
112
* Variables:
113
  - filename : name of the file read in without the path.
114
  - location  : path to the file
115
  - sections : user argument which sections should be stored.
116
  - content : property that gives you the stored OrderedDict
117
  - ID : unique identification of this file (defaults to location+filename)
118
* Functions:
119
  - resolve_key_references()
120
  - get_value(key, must_exist=False)
121
 
122
#### 2.1.2 ConfigTree
123
The ConfigTree base class implements the 'tree'-aspect of the set of configuration files. On construction it reads in a collection of ConfigFiles.
124
 
125
* Arguments:
126
  - rootdirs : list of top directories where to search for files
127
  - filename : name of the files to search for. Use '*' as wild-char, e.g. 'hdl_buildset_*.cfg' matches all buildset files.
128
  - sections : optional argument to limit the sections that are read in.
129
* Variables:
130
  - The three arguments are stored as class variables.
131
  - configfiles : returns a dict containing all read in ConfigFile objects.
132
* Functions:
133
  - remove_files_from_tree(files_to_remove)
134
  - limit_tree_to(files_to_keep)
135
  - get_key_values(key, configfiles=None, must_exist=False)
136
  - get_configfiles(key, values=None, user_configfiles=None)
137
 
138
Note: A function ```_factory_constructor``` is used in the main loop to read in each file that matches the filename(mask) argument. The default implementation calls the ConfigFile constructor. Inhereted classes should implement their own ```_factory_constructor```.
139
 
140
### 2.2 Related classes
141
Three kinds of configuration files are currently used in RadioHDL [1].
142
 
143
- hdl_buildset_.cfg
144
- hdl_tool_.cfg
145
- hdlib.cfg
146
 
147
The keys inside these files decide for which flavour a file qualifies. To implement this we created three derived classes that only implement the ID property and they call the base class constructor with their own set of required keys. In a similar way three flavour of configuration trees are implemented with three derived classes from ConfigTree that only implement their own _factory_constructor function.
148
 
149
An HDL project can have many hdllib configuration files. Therefore it is useful to be able to modify collections of files. When modifying files we like the preserve as much of the original file as possible, this includes comments and spatial layout of the file. Since ConfigFile (the only class that reads in the files) discards this kind of information this class cannot be used. Therefore there is a separate class that can read a configuration file: RawConfigFile. Together with RawConfigTree (derived from ConfigTree) it forms the base for bulk modifying configuration files, see Figure 2.
150
 
151
![Figure 2](./raw_config_classes.jpg "raw_config_classes.jpg")
152
Figure 2: Classes that provide access to the raw content of the configuration file to support modifications.
153
 
154
### 2.3 Unit tests
155
The $RADIOHDL_GEAR/core/tests directory contains unit tests to verify the working of the classes.
156
 
157
### 2.4 Utility modify_configfiles
158
As small interactive python program **modify_configfiles** implements a tiny menu system that enables you to execute the modification functions that the RawConfigFile class provides.
159
 
160
### 2.5 Relation with old CommonDictFile
161
In the initial Python2 code of RadioHDL there one large class called CommonDictFile. This class is now obsolete and replaced by the eight classes. For those who were used to work with CommonDictFile: the next table shows the new function names.
162
 
163
|Old CommonDictFile|New ConfigFile, ConfigTree, **modify_configfiles**|
164
|:---|:---|
165
|dicts()                                          |ConfigTree.configfiles|
166
|nof_dicts()                                      |len(ConfigTree.configfiles)|
167
|filePathNames()                                  |ConfigTree.configfiles.keys()|
168
|filePaths()                                      |iterate over ConfigTree.configfiles, useConfigFile.location|
169
|remove_dict_from_list(dict_to_remove)            |ConfigTree.remove_files_from_tree(files_to_remove)|
170
|remove_all_but_the_dict_from_list(dict_to_keep)  |ConfigTree.limit_tree_to(files_to_keep)|
171
|find_all_dict_file_paths(rootDir=None)           |obsolete|
172
|read_all_dict_files(filePathNames=None)          |obsolete|
173
|read_dict_file(filePathName=None)                |ConfigFile(fullFileName)|
174
|write_dict_file(...)                             |interactive modify_configfiles program|
175
|append_key_to_dict_file(...)                     |interactive modify_configfiles program|
176
|insert_key_in_dict_file_at_line_number(...)      |interactive modify_configfiles program|
177
|insert_key_in_dict_file_before_another_key(...)  |interactive modify_configfiles program|
178
|remove_key_from_dict_file(...)                   |interactive modify_configfiles program|
179
|rename_key_in_dict_file(...)                     |interactive modify_configfiles program|
180
|change_key_value_in_dict_file(...)               |interactive modify_configfiles program|
181
|resolve_key_references()                         |ConfigFile.resolve_key_references()|
182
|get_filePath(the_dict)                           |ConfigFile.location|
183
|get_filePathName(the_dict)                       |ConfigFile.location + '/' + ConfigFile.filename|
184
|get_key_values(key, dicts=None, must_exist=False)|ConfigTree.get_key_values(key, configfiles=None, must_exist=False)|
185
|get_key_value(key, the_dict, must_exist=False)   |ConfigFile.get_value(key, must_exist=False)|
186
|get_dicts(key, values=None, dicts=None)          |ConfigTree.get_configfiles(key, values=None, user_configfiles=None)|
187
 
188
---
189
## 3 Initialisation of the environment
190
The RadioHDL package is setup using **init_radiohdl.sh**. This **init_radiohdl.sh** keeps the shell environment as clean as possible. By not cluttering your environment with many functions (actually everything in **generic.sh**) **init_radiohdl.sh** defines only three environment variables and extends your path with the necessary paths.
191
 
192
### 3.1 Usage of configuration file settings in a shell
193
Using the configuration files is easy since they can be accessed through ConfigFile and ConfigTree. But the content of the configuration files should also be available for shell. The cleanest way to do this is to reuse/wrap the python code so that we don't have to reimplement the file interpretation. So we made two small python programs that read in a configuration file and print the requested information. Two other small shell scripts invoke those python scripts and execute the information that was printed by the python script.
194
 
195
![Figure 3](./sd_export_variable.jpg "sd_export_variable.jpg")
196
Figure 3: Example how configuration file information is made available in shell.
197
 
198
### 3.2 set_quartus.sh and set_modelsim.sh
199
Both scripts only use information from the hdl_buildset- and hdl_tool- configuration files.
200
 
201
---
202
## 4 Programming principles
203
This chapter describes some principles that were used for designing and writing the scripts. In general we can state that:
204
 
205
- scripts must give a syntax help message when they are invoked the wrong way or when '-h' or '--help' is given as an argument
206
- invocation arguments are strictly checked. Unknown arguments result in an error (and the help
207
message)
208
- the order of the invocation arguments is trivial
209
- everything is assumed to be fault/wrong/undefined until the opposite is proven.
210
 
211
### 4.1 Shell scripts
212
#### 4.1.1 Shell options
213
Each shell scripts starts with the line:
214
 
215
```bash
216
#!/bin/bash -eu
217
```
218
where
219
```
220
    -e option: exit immediate on error
221
    -u option: treat undefined variables and parameters as an error
222
```
223
The -u option helps us to find uninitialized variables but also makes it harder to use the invocation arguments: MY_VAR=$1 exits the script with an ugly error message if no arguments were used. Also trying to test the arguments like if [ "$1" == "something" ] will exit the script. However you can access a probably-undefined variable, say VARNAME with ${VARNAME:-} with triggering the -u option.
224
 
225
All scripts nowadays expect the buildset name to be the first argument so the following code snippet catches the undefined first argument is a proper way.
226
 
227
```bash
228
BUILDSET=${1:-}
229
if [ "${BUILDSET}" = "" ]; then
230
  hdl_error $0 "Please specify all arguments\nUsage: $0 "
231
fi
232
```
233
 
234
#### 4.1.2 generic.sh
235
One of the first lines in each script is:
236
```
237
#read generic functions
238
. ${RADIOHDL_GEAR}/generic.sh
239
```
240
This imports some generic functions like path_add, hdl_exec, hdl_exit, and so on. By importing this in each script the environment of the user stays clean and we import the functions only when we need them.
241
 
242
#### 4.1.3 Parsing arguments
243
Unlike Python, there is no out of the box argument parser for shell programming that works well. There are two flavours: **```getopts```** and **```getopt```**.
244
 
245
##### 4.1.3.1 getopts
246
The ```getopts``` argument parser only accepts short options like ```-e something``` or ```-v```. The major flaw of ```getopts``` however is that *options should always precede the arguments*. For example if we have a script **getopts_test.sh** like:
247
 
248
```
249
EXT=
250
VERBOSE=false
251
while getopts e:v option
252
do
253
    case "$option" in
254
        e) EXT=${OPTARG} ;;
255
        v) VERBOSE=true ;;
256
       \?) echo "OOPS"; exit 1 ;;
257
    esac
258
done
259
shift $(($OPTIND - 1))
260
echo "EXT="$EXT
261
echo "VERBOSE="$VERBOSE
262
echo "POSITIONALS=$@"
263
```
264
then
265
```
266
getopts_test.sh -v -e something cats and dogs
267
```
268
will give you the correct output:
269
```
270
EXT=something
271
VERBOSE=true
272
POSITIONALS=cats and dogs
273
```
274
But
275
```
276
getopts_test.sh -v cats and dogs -e something
277
```
278
*silently* treats the -e as positional argument:
279
```
280
EXT=
281
VERBOSE=true
282
POSITIONALS=cats and dogs -e something
283
```
284
 
285
###### 4.1.3.2 getopt
286
The ```getopt``` argument parser is not picky about order of options and arguments and even also excepts long options like ```--extension=something``` or ```--verbose```. Short and long options can be mixed and are recognized by the number is minus signs. This is where it goes wrong! When the user makes a type like ```-extension=something``` (one minus instead of two) it sees short option ```-e``` with the value ```xtension=something```. With a test script **getopt_test.sh**:
287
 
288
```
289
EXT=
290
VERBOSE=false
291
eval set -- `getopt -o e:v --long extension:,verbose -n $0 -- "$@"`
292
while true ; do
293
    case "$1" in
294
        -e|--extension)
295
            EXT="${2:+$2}"
296
            shift 2
297
            ;;
298
        -v|--verbose)
299
            VERBOSE=true
300
            shift
301
            ;;
302
        --) shift ; break ;;
303
        \?) echo "OOPS"; exit 1 ;;
304
        *) echo "Internal error!"; exit 1 ;;
305
    esac
306
done
307
echo "EXT="$EXT
308
echo "VERBOSE="$VERBOSE
309
echo "POSITIONALS=$@"
310
```
311
then
312
```
313
getopt_test.sh -v -e something cats and dogs
314
```
315
will give you the correct output:
316
```
317
EXT=something
318
VERBOSE=true
319
POSITIONALS=cats and dogs
320
```
321
also invocations like
322
```
323
getopt_test.sh --verbose -e something cats and dogs
324
getopt_test.sh -v --extension=something cats and dogs
325
getopt_test.sh -v cats and dogs -e something
326
getopt_test.sh --verbose cats and dogs --extension=something
327
```
328
will all give the correct result.
329
But
330
```
331
getopt_test.sh --verbose cats and dogs -extension=something
332
```
333
*silently* treats the typo of ```-extension``` and give the following result:
334
```
335
EXT=xtension=something
336
VERBOSE=true
337
POSITIONALS=cats and dogs
338
```
339
 
340
##### 4.1.3.3 Chosen solution
341
Since the two out of the box tools both have major flaws we have to make a do-it-yourself (DIY) parser. After extensive research on the internet the following solution was made that is fully correct and is as tiny as possible.
342
```
343
missing_option_argument() {
344
    exit_with_error "Option $1 expects an argument"
345
}
346
 
347
exit_with_error() {
348
    echo "$@"
349
    cat <<@EndOfHelp@
350
Usage: $(basename $0) [options] arguments
351
Options
352
-e | --extension=   
353
-v | --verbose      
354
Arguments
355
356
@EndOfHelp@
357
    exit 1
358
}
359
 
360
POSITIONAL=()
361
EXT=
362
VERBOSE=false
363
while [[ $# -gt 0 ]] do
364
    case $1 in
365
        -e)
366
            [ $# -lt 2 ] && missing_option_argument $1
367
            EXT="$2" ; shift ;;
368
        --extension=*)
369
            EXT=${1#*=} ;;
370
        -v|--verbose)
371
            VERBOSE=true ;;
372
        -h|--help)
373
            exit_with_error "Information about the options and arguments" ;;
374
        -*|--*)
375
            exit_with_error "Unknown option: "$1 ;;
376
        *)
377
            POSITIONAL+=("$1") ;;
378
    esac
379
    shift
380
done
381
if [ ${#POSITIONAL[@]} -gt 0 ]; then
382
    set -- "${POSITIONAL[@]}"
383
fi
384
echo "EXT="$EXT
385
echo "VERBOSE="$VERBOSE
386
echo "POSITIONALS=$@"
387
```
388
To give neat responses to the user when something goes wrong we defined two small functions:
389
 
390
- ```missing_option_argument()``` tells the user that a value is expected for the option and then calls the exit_with_error function.
391
- ```exit_with_error()``` shows the user the correct syntax of the command and exits the script with exitcode 1. Please provide useful information to the user.
392
 
393
The main loop of the parser is only slightly larger than with the out-of-the-box-with-major-flaws parsers. The main idea behind the parser loop is:
394
 
395
1.  handle all defined options. Options without an argument can be combine in one 'case' match. For options that do need an argument we have to treat the short and the long version separate as the short version covers two arguments (no connecting '=' sign) and the long version includes the value of the option.
396
2. catch 'help' options
397
3. reject all other options
398
4. gather the arguments that may be anywhere in the invocation order.
399
 
400
Finally assign the collected positional arguments to $1, $2, and so on.
401
 
402
This DIY parser meets all programming principles we defined in the beginning of this chapter.
403
 
404
### 4.2 Python
405
#### 4.2.1 Importing packages
406
When you need only a few functions from a package you can better limit the import to these few functions. This keeps the 'lookup tables' of python smaller, makes the code cleaner and give insight in what you use from the packages. So instead of writing:
407
```
408
import os.path
409
dir_name=os.path.expandvars('RADIOHDL_GEAR')
410
```
411
write:
412
```
413
from os.path import expandvars
414
dir_name=expandvars('RADIOHDL_GEAR')
415
```
416
 
417
#### 4.2.2 Limiting export
418
If a source file contains both public functions/classes as well as private ones you can limit what a user will see if it imports your file by defining the __all__ variable. E.g. by adding the line:
419
```
420
__all__ = [ 'public_function_1', 'public_class_1', 'public_constant' ]
421
```
422
to your source file limits the exposure the these three entities when someone imports your file.
423
 
424
#### 4.2.3 Parsing arguments
425
Fortunately python has an excellent parser for arguments: ```ArgumentParser``` from the argparse package. Look on internet for the manual or look e.g. in **export_config_variables.py** how to use this parser. In short:
426
 
427
1. create an ```ArgumentParser``` instance.
428
2. for each argument and for each option call ```add_argument```
429
3. finally call ```parse_args()```

powered by: WebSVN 2.1.0

© copyright 1999-2024 OpenCores.org, equivalent to Oliscience, all rights reserved. OpenCores®, registered trademark.