Subversion Repositories radiohdl

# RadioHDL Gear Programmer Guide

### *Speed up HDL development*

---
#### Document history:
|Revision|Date|Author|Affiliation|Modification|
|:---|:---|:---|:---|:---|
| 0.9|18 sep 2018|R. Overeem |ASTRON|HDL programmers manual, created after refactoring the Python3 and bash code.|
| 1.0|27 jan 2020|E. Kooistra|ASTRON|Converted HDL programmers manual docx into this md file|

---
#### Contents:
1 Introduction  
1.1 Purpose  
1.2 Design principles  
1.2.1 Rules for scripts and configuration files  
1.2.2 Rules for keys and values  
2 Handling configuration files  
2.1 Base classes  
2.1.1 ConfigFile  
2.1.2 ConfigTree  
2.2 Related classes  
2.3 Unit tests
2.4 Utility modify_configfiles  
2.5 Relation with old CommonDictFile  
3 Initialisation of the environment  
3.1 Usage of configuration file settings in a shell  
3.2 set_quartus.sh and set_modelsim.sh  
4 Programming principles  
4.1 Shell scripts  
4.1.1 Shell options   
4.1.2 generic.sh  
4.1.3 Parsing arguments  
4.2 Python  
4.2.1 Importing packages  
4.2.2 Limiting export  
4.2.3 Parsing arguments   

---
#### References:
[1] radiohdl_user_guide.md  
[2] radiohdl_hdl_buildset_key_descriptions.md  
[3] radiohdl_hdl_buildset_uniboard1.md  
[4] radiohdl_hdl_tool_modelsim.md  
[5] radiohdl_hdl_tool_quartus.md  
[6] radiohdl_hdl_library_key_descriptions.md  

---
## 1 Introduction
### 1.1 Purpose
This document describes the inside gear of the RadioHDL package [1]. 

### 1.2 Design principles
Originally the RadioHDL gear consisted of Python2 and bash. The conversion to Python3 was taken as an opportunity to refactor the RadioHDL code. These design principles also apply to future updates and extensions of the RadioHDL package, see also section 4.

#### 1.2.1 Rules for scripts and configuration files
* Keep the shell environment as clean as possible  
Therefore RadioHDL uses **init_radiohdl.sh**, to (manually) start RadioHDL in a terminal.

* Store all 'user tuneable' variables in configuration files instead of in scripts  
This avoids too much tool knowledge in scripts and eases the maintenance of the scripts.

* The scripts must clearly report if they fail  
For example the Python scripts give a hint of what causes the failure. Bash scripts report if they fail, instead of finishing silently.

* Avoid classes that can do too much  
Instead Use classes that 'embed/guard' some data(structure) and provide a few functions for manipulating this data in a controlled way. This normally means that a class is very good in only one or two things. The maintenance of multiple dedicated classes is much easier, since it is very clear what each class does. This rule is applied in section 2.

#### 1.2.2 Rules for keys and values
* Source or target oriented keys  
Whether a key is source oriented or target oriented depends on whether its files are used for one or more targets. In general if a file is used for more targets then source oriented is preferred to avoid having to list the file name twice. If a file is used only for one target then target oriented is preferred to be more clear about the purpose of the key. For example the quartus_* keys in the hdllib.cfg [5], [6] are now source oriented. Instead it may be better to redefine them as target oriented. E.g. a 'quartus_create_qsf' key that defines to create a qsf file using the information listed in the values. 

* Avoid hidden behaviour of keys  
The 'synth_top_level_entity' key in the hdllib.cfg [6] enforces the creation of a qpf and qsf. This kind of hidden behaviour is not so nice.  Instead it is more clear to have an explicit 'quartus_create_qpf' and 'quartus_create_qsf' key to define this.  

* Support new key names  
Unknown key names are ignored, such that multiple tool scripts can use the same cfg file. Each tool only handles the keys that it knowns. Hence a key name will only cause an action if it is known by a tool script. Therefore key names must be predefined and can be described in a tool configuration file specification document.

* Support more value definitions for keys  
Currently keys can have one value, a list of values or a list of pairs [1]. More value definitions could be added to the RadioHDL configuration file schema, e.g. a list of value tuples, whereby a tuple can contain more than a pair. 

* Executing key values  
An issue can be that it can be dangerous to blindly execute a key value, because it is user defined and could be a script that contains e.g. 'rm -rf ~/*'.  E.g. 'quartus_tcl_files' sources a tcl script, it is left to the user to ensure that this tcl script is a legal tcl script. For shell commands blind execution of commands can be prevented by defining a dedicated key per command, such that it is impossible to execute a key value, so instead of e.g. 'ls *' the key value then become '*' and the key name itself invokes 'ls'.

* Section headers  
Support for new tools may be added by using a dedicated [section header] in the configuration file.

---
## 2 Handling configuration files 
The handling of the configuration files involves e.g.:

- represent the content of one configuration file 
- represent the content of a whole hierarchical tree of configuration files 
- a bulk modification tool to modify keys and/or values

This diverse functionality has resulted in eight  classes and one interactive command line tool. Figure 1 shows the class diagram. 

![Figure 1](./configuration_file_classes.jpg "configuration_file_classes.jpg")  
Figure 1: Class diagram of the configuration file classes.

### 2.1 Base clases
The main functionality located in two base classes *ConfigFile* and *ConfigTree*.

#### 2.1.1 ConfigFile
The ConfigFile base class can read in one configuration file and store that content to an OrderedDict inside the class. This is done during the construction of the class. If the read in fails a ConfigFileException is thrown.

* Arguments:
  - filename : full filename including the absolute path of the file 
  - sections : optional argument to limit the sections that are read in. 
  - required_keys : optional list of key names that must exist in the configuration file. 
* Variables:
  - filename : name of the file read in without the path. 
  - location  : path to the file 
  - sections : user argument which sections should be stored. 
  - content : property that gives you the stored OrderedDict 
  - ID : unique identification of this file (defaults to location+filename) 
* Functions:
  - resolve_key_references() 
  - get_value(key, must_exist=False) 

#### 2.1.2 ConfigTree 
The ConfigTree base class implements the 'tree'-aspect of the set of configuration files. On construction it reads in a collection of ConfigFiles.

* Arguments:
  - rootdirs : list of top directories where to search for files 
  - filename : name of the files to search for. Use '*' as wild-char, e.g. 'hdl_buildset_*.cfg' matches all buildset files. 
  - sections : optional argument to limit the sections that are read in. 
* Variables:
  - The three arguments are stored as class variables. 
  - configfiles : returns a dict containing all read in ConfigFile objects. 
* Functions:
  - remove_files_from_tree(files_to_remove) 
  - limit_tree_to(files_to_keep) 
  - get_key_values(key, configfiles=None, must_exist=False) 
  - get_configfiles(key, values=None, user_configfiles=None) 

Note: A function ```_factory_constructor``` is used in the main loop to read in each file that matches the filename(mask) argument. The default implementation calls the ConfigFile constructor. Inhereted classes should implement their own ```_factory_constructor```. 

### 2.2 Related classes 
Three kinds of configuration files are currently used in RadioHDL [1].

- hdl_buildset_<buildset_name>.cfg 
- hdl_tool_<tool_name>.cfg 
- hdlib.cfg 

The keys inside these files decide for which flavour a file qualifies. To implement this we created three derived classes that only implement the ID property and they call the base class constructor with their own set of required keys. In a similar way three flavour of configuration trees are implemented with three derived classes from ConfigTree that only implement their own _factory_constructor function.

An HDL project can have many hdllib configuration files. Therefore it is useful to be able to modify collections of files. When modifying files we like the preserve as much of the original file as possible, this includes comments and spatial layout of the file. Since ConfigFile (the only class that reads in the files) discards this kind of information this class cannot be used. Therefore there is a separate class that can read a configuration file: RawConfigFile. Together with RawConfigTree (derived from ConfigTree) it forms the base for bulk modifying configuration files, see Figure 2. 

![Figure 2](./raw_config_classes.jpg "raw_config_classes.jpg")  
Figure 2: Classes that provide access to the raw content of the configuration file to support modifications. 

### 2.3 Unit tests
The $RADIOHDL_GEAR/core/tests directory contains unit tests to verify the working of the classes.

### 2.4 Utility modify_configfiles
As small interactive python program **modify_configfiles** implements a tiny menu system that enables you to execute the modification functions that the RawConfigFile class provides. 

### 2.5 Relation with old CommonDictFile 
In the initial Python2 code of RadioHDL there one large class called CommonDictFile. This class is now obsolete and replaced by the eight classes. For those who were used to work with CommonDictFile: the next table shows the new function names.

|Old CommonDictFile|New ConfigFile, ConfigTree, **modify_configfiles**|
|:---|:---|
|dicts()                                          |ConfigTree.configfiles|
|nof_dicts()                                      |len(ConfigTree.configfiles)| 
|filePathNames()                                  |ConfigTree.configfiles.keys()|
|filePaths()                                      |iterate over ConfigTree.configfiles, useConfigFile.location|
|remove_dict_from_list(dict_to_remove)            |ConfigTree.remove_files_from_tree(files_to_remove)|
|remove_all_but_the_dict_from_list(dict_to_keep)  |ConfigTree.limit_tree_to(files_to_keep)|
|find_all_dict_file_paths(rootDir=None)           |obsolete|
|read_all_dict_files(filePathNames=None)          |obsolete| 
|read_dict_file(filePathName=None)                |ConfigFile(fullFileName)|
|write_dict_file(...)                             |interactive modify_configfiles program|
|append_key_to_dict_file(...)                     |interactive modify_configfiles program|
|insert_key_in_dict_file_at_line_number(...)      |interactive modify_configfiles program|
|insert_key_in_dict_file_before_another_key(...)  |interactive modify_configfiles program|
|remove_key_from_dict_file(...)                   |interactive modify_configfiles program|
|rename_key_in_dict_file(...)                     |interactive modify_configfiles program|
|change_key_value_in_dict_file(...)               |interactive modify_configfiles program|
|resolve_key_references()                         |ConfigFile.resolve_key_references()|
|get_filePath(the_dict)                           |ConfigFile.location|
|get_filePathName(the_dict)                       |ConfigFile.location + '/' + ConfigFile.filename|
|get_key_values(key, dicts=None, must_exist=False)|ConfigTree.get_key_values(key, configfiles=None, must_exist=False)|
|get_key_value(key, the_dict, must_exist=False)   |ConfigFile.get_value(key, must_exist=False)|
|get_dicts(key, values=None, dicts=None)          |ConfigTree.get_configfiles(key, values=None, user_configfiles=None)|

---
## 3 Initialisation of the environment 
The RadioHDL package is setup using **init_radiohdl.sh**. This **init_radiohdl.sh** keeps the shell environment as clean as possible. By not cluttering your environment with many functions (actually everything in **generic.sh**) **init_radiohdl.sh** defines only three environment variables and extends your path with the necessary paths.

### 3.1 Usage of configuration file settings in a shell
Using the configuration files is easy since they can be accessed through ConfigFile and ConfigTree. But the content of the configuration files should also be available for shell. The cleanest way to do this is to reuse/wrap the python code so that we don't have to reimplement the file interpretation. So we made two small python programs that read in a configuration file and print the requested information. Two other small shell scripts invoke those python scripts and execute the information that was printed by the python script. 

![Figure 3](./sd_export_variable.jpg "sd_export_variable.jpg")  
Figure 3: Example how configuration file information is made available in shell. 

### 3.2 set_quartus.sh and set_modelsim.sh
Both scripts only use information from the hdl_buildset- and hdl_tool- configuration files. 

---
## 4 Programming principles 
This chapter describes some principles that were used for designing and writing the scripts. In general we can state that:

- scripts must give a syntax help message when they are invoked the wrong way or when '-h' or '--help' is given as an argument
- invocation arguments are strictly checked. Unknown arguments result in an error (and the help 
message)
- the order of the invocation arguments is trivial
- everything is assumed to be fault/wrong/undefined until the opposite is proven. 

### 4.1 Shell scripts 
#### 4.1.1 Shell options 
Each shell scripts starts with the line:

```bash
#!/bin/bash -eu  
```
where
```
    -e option: exit immediate on error  
    -u option: treat undefined variables and parameters as an error  
```
The -u option helps us to find uninitialized variables but also makes it harder to use the invocation arguments: MY_VAR=$1 exits the script with an ugly error message if no arguments were used. Also trying to test the arguments like if [ "$1" == "something" ] will exit the script. However you can access a probably-undefined variable, say VARNAME with ${VARNAME:-} with triggering the -u option.
 
All scripts nowadays expect the buildset name to be the first argument so the following code snippet catches the undefined first argument is a proper way.

```bash
BUILDSET=${1:-}
if [ "${BUILDSET}" = "" ]; then
  hdl_error $0 "Please specify all arguments\nUsage: $0 <buildset>"
fi
```

#### 4.1.2 generic.sh
One of the first lines in each script is: 
```
#read generic functions
. ${RADIOHDL_GEAR}/generic.sh 
```
This imports some generic functions like path_add, hdl_exec, hdl_exit, and so on. By importing this in each script the environment of the user stays clean and we import the functions only when we need them. 

#### 4.1.3 Parsing arguments 
Unlike Python, there is no out of the box argument parser for shell programming that works well. There are two flavours: **```getopts```** and **```getopt```**. 

##### 4.1.3.1 getopts 
The ```getopts``` argument parser only accepts short options like ```-e something``` or ```-v```. The major flaw of ```getopts``` however is that *options should always precede the arguments*. For example if we have a script **getopts_test.sh** like: 

```
EXT= 
VERBOSE=false
while getopts e:v option
do
    case "$option" in 
        e) EXT=${OPTARG} ;;
        v) VERBOSE=true ;;
       \?) echo "OOPS"; exit 1 ;; 
    esac
done
shift $(($OPTIND - 1))
echo "EXT="$EXT
echo "VERBOSE="$VERBOSE
echo "POSITIONALS=$@" 
```
then 
```
getopts_test.sh -v -e something cats and dogs 
```
will give you the correct output: 
```
EXT=something
VERBOSE=true
POSITIONALS=cats and dogs 
```
But 
```
getopts_test.sh -v cats and dogs -e something 
```
*silently* treats the -e as positional argument: 
```
EXT=  
VERBOSE=true
POSITIONALS=cats and dogs -e something 
```

###### 4.1.3.2 getopt 
The ```getopt``` argument parser is not picky about order of options and arguments and even also excepts long options like ```--extension=something``` or ```--verbose```. Short and long options can be mixed and are recognized by the number is minus signs. This is where it goes wrong! When the user makes a type like ```-extension=something``` (one minus instead of two) it sees short option ```-e``` with the value ```xtension=something```. With a test script **getopt_test.sh**:

```
EXT= 
VERBOSE=false 
eval set -- `getopt -o e:v --long extension:,verbose -n $0 -- "$@"` 
while true ; do
    case "$1" in 
        -e|--extension) 
            EXT="${2:+$2}" 
            shift 2
            ;;
        -v|--verbose) 
            VERBOSE=true 
            shift
            ;;
        --) shift ; break ;; 
        \?) echo "OOPS"; exit 1 ;; 
        *) echo "Internal error!"; exit 1 ;; 
    esac
done
echo "EXT="$EXT
echo "VERBOSE="$VERBOSE
echo "POSITIONALS=$@" 
```
then
```
getopt_test.sh -v -e something cats and dogs 
```
will give you the correct output: 
```
EXT=something  
VERBOSE=true  
POSITIONALS=cats and dogs  
```
also invocations like 
```
getopt_test.sh --verbose -e something cats and dogs
getopt_test.sh -v --extension=something cats and dogs
getopt_test.sh -v cats and dogs -e something
getopt_test.sh --verbose cats and dogs --extension=something 
```
will all give the correct result. 
But 
```
getopt_test.sh --verbose cats and dogs -extension=something 
```
*silently* treats the typo of ```-extension``` and give the following result: 
```
EXT=xtension=something
VERBOSE=true
POSITIONALS=cats and dogs 
```

##### 4.1.3.3 Chosen solution
Since the two out of the box tools both have major flaws we have to make a do-it-yourself (DIY) parser. After extensive research on the internet the following solution was made that is fully correct and is as tiny as possible.
```
missing_option_argument() {
    exit_with_error "Option $1 expects an argument"
}

exit_with_error() {
    echo "$@"
    cat <<@EndOfHelp@ 
Usage: $(basename $0) [options] arguments 
Options 
-e | --extension=   <explain> 
-v | --verbose      <explain> 
Arguments 
<explain> 
@EndOfHelp@
    exit 1
} 

POSITIONAL=()
EXT=
VERBOSE=false
while [[ $# -gt 0 ]] do
    case $1 in 
        -e) 
            [ $# -lt 2 ] && missing_option_argument $1 
            EXT="$2" ; shift ;;
        --extension=*) 
            EXT=${1#*=} ;; 
        -v|--verbose) 
            VERBOSE=true ;;
        -h|--help) 
            exit_with_error "Information about the options and arguments" ;; 
        -*|--*) 
            exit_with_error "Unknown option: "$1 ;; 
        *) 
            POSITIONAL+=("$1") ;;
    esac 
    shift
done
if [ ${#POSITIONAL[@]} -gt 0 ]; then 
    set -- "${POSITIONAL[@]}"
fi
echo "EXT="$EXT
echo "VERBOSE="$VERBOSE
echo "POSITIONALS=$@" 
```
To give neat responses to the user when something goes wrong we defined two small functions:

- ```missing_option_argument()``` tells the user that a value is expected for the option and then calls the exit_with_error function.
- ```exit_with_error()``` shows the user the correct syntax of the command and exits the script with exitcode 1. Please provide useful information to the user.

The main loop of the parser is only slightly larger than with the out-of-the-box-with-major-flaws parsers. The main idea behind the parser loop is:
 
1.  handle all defined options. Options without an argument can be combine in one 'case' match. For options that do need an argument we have to treat the short and the long version separate as the short version covers two arguments (no connecting '=' sign) and the long version includes the value of the option. 
2. catch 'help' options 
3. reject all other options 
4. gather the arguments that may be anywhere in the invocation order. 

Finally assign the collected positional arguments to $1, $2, and so on.

This DIY parser meets all programming principles we defined in the beginning of this chapter. 

### 4.2 Python
#### 4.2.1 Importing packages
When you need only a few functions from a package you can better limit the import to these few functions. This keeps the 'lookup tables' of python smaller, makes the code cleaner and give insight in what you use from the packages. So instead of writing: 
```
import os.path
dir_name=os.path.expandvars('RADIOHDL_GEAR') 
```
write: 
```
from os.path import expandvars
dir_name=expandvars('RADIOHDL_GEAR') 
```

#### 4.2.2 Limiting export
If a source file contains both public functions/classes as well as private ones you can limit what a user will see if it imports your file by defining the __all__ variable. E.g. by adding the line:
```
__all__ = [ 'public_function_1', 'public_class_1', 'public_constant' ] 
```
to your source file limits the exposure the these three entities when someone imports your file. 

#### 4.2.3 Parsing arguments 
Fortunately python has an excellent parser for arguments: ```ArgumentParser``` from the argparse package. Look on internet for the manual or look e.g. in **export_config_variables.py** how to use this parser. In short:

1. create an ```ArgumentParser``` instance. 
2. for each argument and for each option call ```add_argument``` 
3. finally call ```parse_args()```