Week 5: Expose API and API tests

The API!

As planned for week 5, the work on proper exposure of API has been completed with the help of thorough guidance by the mentors. Moreover, the modules have been separated among core, cli, tools, and templates. Here is the link to the forked repository working branch. Moreover, the modtool tests for the core modules newmod, add, rename, rm (remove) have been written. Here is the link to the GitHub repository.


Regarding the core modules, the setup function has been removed and all the arguments get their values in the __init__ itself. Moreover, proper checks like check on boolean values and improper arguments have been made in a separate validate function which does not run for arguments passed through CLI because of the checks in the CLI modules themselves. The checks in the CLI modules are same as the ones that existed earlier and have not been moved to the core modules considering the fact that the code should break just after an incorrect argument is passed rather than breaking after all the arguments have been passed for the convenience of the user. Thus, validation for CLI commands does not take place twice. The common parameters get initialized in the  __init__ function of the base module class Modtool. Moreover, the ModTool.__init__ does not run completely for commands newmod and info because that is not necessary and also assumes an existing module. So, if type(self).__name__ in ['ModToolInfo', 'ModToolNewModule'] the code breaks after initializing the arguments. All the imports for module classes or functional definitions have been made relative. Some of the protected members of the modtool classes have been made public for the access of CLI modules and for the simplicity of the user.

The tools like code_generator, grc_xml_generator​ have been moved to a separate directory tools which is accessible to both CLI and Core modules (the CLI modules utilize some of the util_functions). The templates, gr_newmod and templates.py, have been moved to a separate templates directory. The prefix modtool_ has been removed from the core modules since the helper modules have been moved to a separate directory.

Modtool Tests

The modtool testing begins with the creation of a new temporary directory in setUpClass function which runs before the actual testing begins and ends with the deletion of that directory in tearDownClass function which runs after the entire testing is complete.

Before test for any module, a setUp function runs the commands for creation of new module in the temporary directory and then adds a block if the previous command ran without raising an exception. After that, the specific tests for the module are run. The tests are skipped if the command necessary for their execution has already failed (like remove will be skipped if add fails in the setUp​).

The tests for the API classes ModToolNewModule, ModToolAdd, ModToolRename, and ModToolRemove have been written. The testing includes the raising of particular exceptions for class instantiation and the assertion for the existence or non-existence of a particular directory or module. For the class ModToolRemove, there are tests for asserting that the incorrect files are not removed. The reason for the exceptions has been mentioned before most of the assertRasises statements as a comment.

Moreover, I found a redundant patch of code in the class ModToolRemove of the master branch. Here is the link to the pull request.

Tasks for Week 6

  1. Enhance the tests and add the source in-tree.
  2. Bug fixes for the API (if any).
  3. Add a logger and use it instead of print statements.
  4. Split long functions.
  5. Make the nested functions without the need of closure as global.

Link to forked repository working branch (core_cli)

Link to the repository containing modtool tests.

Link to the updated GitHub project.

Week 4: Core/CLI split

The Core and the CLI splits!

As planned for week 4, the Core and the CLI portions of the modtool have been separated so the modtool becomes scriptable from python scripts. The work is done on the forked repository module_sep branch. Moreover, I have also worked on the enhancement of python3 branch.

The CLI portion of the modtool still holds the same functionality for all the commands. The entire click portion and the user CLI interaction is placed in the CLI modules whereas the core functionality of modtool that can be exposed as an API is in the Core modules. All the print statements in the core are only executed if the user has used the command through the command line interface. Moreover, since the setup function of the core becomes redundant for the CLI calls, it will only be executed for the API calls. The entire code has been thoroughly tested.

Currently, the user needs to call the run function from the imported module with the options and arguments passed as a dictionary. For example:-

>>> from gnuradio.modtool.core import *
>>> ModToolNewModule().run({‘directory’:’/home/swapnil/Documents/modtool’, ‘module_name’:’square_ff’})
>>> ModToolAdd().run({‘blockname’:’test_block’, ‘directory’:’/home/swapnil/Documents/modtool/gr-square_ff’, ‘lang’:’cpp’, ‘block_type’:’general’, ‘add_cpp_qa’: True})

works absolutely awesome.

Similarly, all the commands except for gr_modtool info work from the python scripts.

There is a slight problem with the ModToolException as it is giving traceback for the exceptions raised.

There was a problem with the grc_xml_generator for python3 compatibility. Here is the link to the pull request.
Here is the link to another enhancement pull request for modtool_newmod.py.

Tasks for Week 5

  • Bug fixes and enhancement of the current work on the module_sep branch
  • Completely expose the API for the modtool
  • Complete modtool testing for the API

Link to forked repository module_sep branch.

Link to the updated Github project.

Week 3: Enhancement, Documentation, Separation of Modules (started)

The first major pull request!

As planned for Week 3, the modtool has been rewritten as plug-in architecture with the external plug-ins working absolutely fine (thanks to awesome guidance by the mentors). The pull request has been made from the forked repository modtool branch. I have also worked on documenting the click plug-in architecture. Here is the link to the documentation. I have also started working on separation of modules on the forked repository module_sep branch.

Initially, I worked on making the external plug-ins work for the modtool. The only mistake that I was committing was I was installing the external plug-in inside a virtual environment so the modtool was not able to load the package through the entry_point (I tried several things to make that work though). After that, I worked on refactoring the code and also made the suggested changes by the mentors.

The documentation part was an awesome experience. It was my first experience on writing a technical blog.

The unit-testing portion also seems exciting. In between, I slightly learned about writing  unit tests, including setup and teardown functions as well as classes that can be beneficial for modtool testing (I slightly read about mocking too, not really beneficial here!).

The work on separation of modules is already in progress. I have already separated the entire click portion from the core without changing the current functionality of the modtool. Now, I need to shift the CLI portion of various other functions so that the core remains unaware of the CLI.

There were slight problems with the cmake, the modified code was not being updated at the dist-packages directory from where it was called during execution on running make and make install. Actually, I never specified the prefix while re-building GNU Radio. The problem was sorted out with the guidance of the mentors.

Tasks for Week 4

  1.  Bug fixes of the current modtool branch, if any.
  2.  Work on separation of modules (major task).
  3.  Work on modtool testing.

Link to the updated Github Project.

Link to the forked repository working branch (separation of modules).

Link to the forked repository modtool branch (plug-in stuff).

Week 2: Plug-in Architecture for modtool

The awesome week with lots of learning experience.

As planned for the Week 2, gr_modtool’s CLI has been re-written as a plug-in architecture. Now the cli()  function of modtool_base.py is called whenever gr_modtool is referenced on the command line. If no further command is mentioned, it shows the help page with available modtool commands. If a command is also given like gr_modtool add, the cli() function of the respective plugin is called.

I have also worked on adding space for external plugins where I rewrote functions of the class click.Group() to achieve the functionality of external plugins along with an internal plug-in architecture for ease of adding modules in-tree. The external plug-in thing uses iter_entry_points for accessing the main module. The only problem that I am facing it that it does not recognize gnuradio.modtool package so I will have read about the implementation of iter_entry_points. The same has been tested creating a modtool_base.py  locally, installing it with the setuptools and changing the directory path from where the plug-ins are loaded. The latter works absolutely fine with the external plug-ins since the package name is recognized correctly.

All the bugs of the CLI have been fixed and it is most likely that I will submit a pull request will the external plugins working by the next week.

Moreover, I have also worked on SequenceCompleter for python3 branch this week. Here is the pull request for the same.

Pull request 1788 has been merged to the python3 branch.

Tasks for Week 3

  1. Fix the issue with loading external plug-ins.
  2. Work on separating the CLI portion from the core Modtool classes, i.e., the modtool directory will be split into two sub-directories, one with the core modtool classes and other with the click CLI code.
  3. Start working on Modtool tests which will test basic syntax errors, whether a required function is called with the required number of arguments, etc.
  4. Work on documenting the current Modtool architecture and the methodology for extending external plug-ins possibly along with a video tutorial.

Link to forked repository’s working branch.

Link to the updated Github project.

Week 1: Move from Argparse to Click

The GSoc’18 Coding period begins!

As planned for the Week 1, the CLI parsing of the modtool has been moved from Argparse to Click on the forked repository’s modtool branch. Moreover, I have also read about the existing plug-in architectures with Click and discussed the same with the mentors.

Click is based on decorators and uses the concept of groups (a type of Multicommand) and commands for the CLI parsing wherein several commands can be registered to a group and both groups and commands can have their specific options (optional) and arguments (generally mandatory, can be made optional).

Currently, no option (or argument) has been registered to the base group because of two main reasons:-

  1. Click does not possess the magical behavior to differentiate whether an option belongs to a group or a command. So, if we register options to the base group, the command will be gr_modtool –skip-lib -t general instead of gr_modtool add -t general –skip-lib.
  2. Some command may not require options like ‘skip-lib’ like gr_modtool info.

But to reduce the redundancy the options common to most of the commands have been registered to the commands as common_params.

In the initial phase of the week, I also faced difficulty in building the ‘python3’ branch of GNU Radio due to multiple installations of log4cpp (most probably) which was rectified with the able guidance of the mentors.

I also found an issue with str.translate(), the pull request has been sent.

Tasks for Week 2

  1. Bug fixes of the Click CLI, if any.
  2. Write the current modtool scripts as a plug-in architecture.
  3. Start working for external plug-ins.
  4. Bug fixes of the python3 branch, if any.

Link to the link to the updated Github project.

Week 0: End of Community Bonding Period

The coding period for GSoC’18 starts on 14th of May. I am highly excited to work on a full-fledged open source project.

I received immense support from the assigned GSoC mentors and the community in the community bonding period. We discussed on various CLI parsing modules along with their advantages and disadvantages, the missing dependencies for GNU Radio “python3” branch in my local machine, templating engines, the workflow for GSoC project, and some general queries.

In the past week, I have gone through the codebase of modtool in the “python3” branch, read about the differences, advantages, and disadvantages of Click over Argparse, read about some of the python API creation frameworks, worked on installing the missing dependencies for the “python3” branch and cherry-picked some of the commits from “next” to my “modtool” branch (still some work to do).

In the beginning, I was missing some of the dependencies for the “python3” branch, including Qt5 and PyQt5 (System detail: Debian 4.14.13-1kali1), which were subsequently installed. But, I am still facing difficulty in compiling the branch as some errors for out-of-scope functional definitions are raised while compiling it. I have opened a Github issue in this regard and I am really looking forward to an explanation.

Tasks for Week 1

  1. Move the CLI parsing to Click: We’ll be shifting to Click for CLI parsing as Click is based on decorators which makes it look much cleaner than argparse. Moreover, Click is fully nestable and works the same with python2 and python3. The basic motivation behind using Click is to ease the process of Plug-in development. This work will lead to no functionality change in the current modtool.
  2. Read about various plug-in managers: I will read about the various existing plug-in managers to have a thorough insight and, if required, use one of them.
  3. Start work on plug-in manager: I’ll start working on the plug-in manager.

  1.  Link to the updated project with week 1 milestones.
  2.  Link to the GitHub issue raised for compilation failure.
  3.  Link for the differences between Docopt, Argparse, and Click.



GSoC’18 Introduction

Google announced the list of accepted projects on 23rd Apr ’18 and I have been accepted as a student developer for GNU Radio. My project “gr-modtool overhaul” revolves around the architectural overhaul of the magical “modtool”,  a script which automates the monotonous work involved in writing the boilerplate code, makefile editing, etc. while creating an Out-Of-Tree module. I am highly excited about the project and am looking forward to contributing as much as possible to the tool. During the entire coding period, I will be mentored by Nicolas Cuervo and Sebastian Koslowski.

I started working on GNU Radio project from February ending and started contributing to it at the beginning of March. Till now I have added the feature for adding a copyright holder to a block, added a list of sensible blocknames for autocompletion while removing or renaming the blocks, created a check for the pre-existence of the same blockname while adding a new block, improved check while removing a block and fixed some of the Pylint issues in the modtool scripts. Moreover, I have also updated the Wiki page “OutOfTreeModules” in compliance with the addition of copyright holder. These tasks thoroughly helped me in understanding how the community works. Here is the link to my GSoC proposal.

I was busy with my examinations till 30th Apr ’18 so the official discussion on the project started on the 30th Apr ’18. We have planned on having a brief chat on Google Hangouts every Friday to discuss the progress of the project and the future tasks and having the blog updated every Saturday to let everyone know about the project and maintain thorough transparency. A mail will also be sent to the GNU Radio mailing list to let people know about the blog updates on Saturday itself.

Currently, I will work on “python3” branch Cherry-picking the necessary commits from the “next” branch. In the forked GNU Radio repository, there will be “modtool” branch which will be the home to regular commits and thus will contain the recent version of the code. Here is the link to the forked repository branches.

Tasks before the coding period

  1. In-depth study of the entire modtool on various branches.
  2. Re-defining milestones and timeline of the project (in accordance with the mentors).
  3. Detailed study of Click and Argparse (python packages for CLI).
  4. A brief study on Python API creation frameworks.

Please feel free to give me any sort of advice or feedback on the structure of the project, the ongoing tasks, or any sort of features that will be an enhancement to the tool.