The evaluation client calls the ROS service ‘group_classification’ for each scene in a supplied JSON scenes file(see here). The perception for each scene is read from a perceptions file (Simulated Perception).
The output of the classification is stored to a flat text file for later processing.
Evaluation client usage¶
Usage: rosrun strands_qsr_evalution evaluation_client.py [options]
Options:
-h, --help show this help message and exit
-p FILE, --perception_filename=FILE
read perceptions from FILE
-s FILE, --scenes=FILE
load the scenes from FILE
-o FILE, --output=FILE
store results in FILE
-O, --overwrite overwrite output file if already exists.
chris@heatwave:~/documents/strands/strands_qsr.wiki$
for example
rosrun strands_qsr_evalution evaluation_client.py -s data/simulation/bham_office_desk_500.json -p simulated_perceptions.json -o run1_results.txt
All object classes that exist in the specified simulated perceptions file will be considered - ie. to test on a subset of the object classes in a given scenes file, create a simulated perception file with only the desired object types.
Output format¶
The results file looks like this:
--
scene1
Pencil Monitor Cup Telephone PC Keyboard
Pencil Laptop Cup Mouse PC Keyboard
Pencil Cup Cup Mouse PC Keyboard
--
scene2
Pencil Monitor Monitor Telephone Bottle Keyboard Pencil
Pencil Monitor Monitor Telephone Bottle Cup Stapler
Pencil Monitor Monitor Telephone Bottle Cup Stapler
--
...
each labelled scene is specified by 5 lines: 1. A separator ‘–’ 1. The scene id as read from the scenes file 1. A tab separated list of ground truth labels 1. A tab separated list of perception labels 1. A tab separated list of relation+perception labels
So, if perception is 100% then lines 3 & 4 will be the same.
Original page: https://github.com/strands-project/strands_qsr/wiki/Evaluation-Client