BuildSys-package {BuildSys}R Documentation

System for Building and Debugging C/C++ Dynamic Libraries

Description

A build system based on 'GNU make' that creates and maintains (simply) make files in an R session and provides GUI debugging support through 'Microsoft Visual Code'.

Details

In the standard R approach dynamic libraries are typically built using the R CMD interface. That interface in turn uses the GNU make system to build the library through a generic makefile bundled with R. This is presumably done to simplify the process of building dynamic libraries for the less experience. The downside however, is adding complexity to the process when doing more esoteric programming that makes use of static or dynamic libraries outside of those bundled with R. In those cases it is likely that we may need to specify compiler include options to be able to successfully compile the code but R CMD SHLIB provides no direct method of acomplishing this task. Instead we have to create a Makevars file that defines the variables PKG_CFLAGS, PKG_CXXFLAGS or PKG_FFLAGS to achieve the desired outcome.

It is also common to have build issues with libraries not linking because of missing dependencies or link order issues (libraries must be linked in the correct order with gcc to ensure error free linking). Resolving these link related issues is made more complex through the R CMD interface because it is no longer possible to simply look at the makefile definition and make modifications to it in an attempt to resolve the build problem. By having a system approach whereby makefiles are the centre of the build process and not obfuscated away in an R installation folder, the process of debugging build issues is made simpler by virtue of the fact that all necessary changes to fix the build are in one file only, and a simple one at that.

Lastly, a major part of developing library code is debugging and standard R provides little support for that. Developers typically rely on gdb / lldb command line driven debugger support, which is daunting for an inexperienced user, so much so that many will simply resort to print statement based debugging approaches, which whilst capable of resolving bugs, is a cumbersome and slow approach. What is required is an easy to use and full featured GUI based debugger and to that end, Microsoft Visual Code coupled with the machine interface (/MI) support in gdb / lldb provide the perfect solution.

BuildSys provides a simple but complete and flexible way to create project makefiles, build libraries and launch GUI based debug sessions, all from within R. The makefiles it creates are permanent rather than transitory so in the event of build troubles it is easy to resort to tinkering with the makefile to resolve the cause of the build problem and then reflect the necessary fixes in the R based project definition.

Project Creation

BuildSys encapsulates a C/C++ dynamic library project definition into an S4 class BSysProject. To create a new project we simply create a new instance of that class and BuildSys will construct the appropriate project definition. For example,

Project <- new("BSysProject", "./MyFolder")

will search the directory ./MyFolder for source files (.c, .cpp etc) and add them to the project definition. In searching for source files it will also scan the found source files for include statements and any found includes will be saved as dependencies for the give source file. No attempt is made to properly parse the source so if the code uses conditional preprocessing statements the situation may arrise that an include file is marked as a dependency when for the given build configuration it may not be. In any case, this will not affect the ability to correctly build the library but will just mean the dependencies are marked wider than true.

Another feature of the dependency check is that any include files not found are considered externalities and if they belong to a known set, will result in the automatic addition of library and include path dependencies for the project. Currently BuildSys knows about TMB.hpp, Rcpp.hpp and RcppEigen.hpp, so if any of these are included into the source file then the necessary include file and define dependencies will be added. This added feature means TMB and Rcpp users have minimalistic usage requirements to construct libraries with BuildSys and need not provide any additional information other than the working folder and possibly the source files being compiled.

A typical project will also need to be given a name but if there is only one source file then the name can be omitted and is inferred from the filename of the source file. For instance, we can explicitly name the project with,

Project <- new("BSysProject", WorkingFolder="./MyFolder", Name="MyProject")

A WorkingFolder must be provided which names the folder where the source code resides and can be either an absolute path or one relative to the R working directory.

Finally, if our project is hierarchical in its structure (eg. having source in one folder, header files in another folder, and binary output in yet another) BuildSys can accomodate that too. In such case we set Flat=FALSE with,

Project <- new("BSysProject", WorkingFolder="./MyFolder", Name="MyProject", Flat=FALSE)

For this case the relevent folders are,

MyFolder/src
MyFolder/include
MyFolder/obj

These sub folder names are the default behaviour. If these sub folder names are not as required they can be renamed using the SourceName, IncludeName and ObjName arguments in the BSysProject initilizer. That is,

Project <- new("BSysProject", WorkingFolder="./MyFolder", Name="MyProject", Flat=FALSE, SourceName="source", IncludeName="header")

Given an existing project, we can also re-initialize it using the initProjectFromFolder method which has the same argument list as the initialize method (it is actually called from the initialize method). For instance, as with the above,

Project <- initProjectFromFolder(Project, WorkingFolder="./MyFolder", Name="MyProject", Flat=FALSE, SourceName="source", IncludeName="header")

Project Compilation

Once we have constructed a Project object compilation is simply a matter of calling the make method. As with traditional GNU makefiles, calling make with no arguments compiles the dynamic library. Calling make with a "clean" argument erases all object files and the built dynamic library. For example,

make(Project) # Compiles the source files and links the dynamic library

make(Project, "clean") # deletes all .o object files and the dynamic library

Since the makefile is permanent a subsequent call to make for an already built project will return immediately as the library is already present and up to date. If a source file is altered then only the files with stale dependencies will be recompiled and linked into a new dynamic library build.

Internally make will call the buildMakefile method to first construct an up to date makefile representing the project. Each makefile has an md5 digest in it which will be updated should you alter the project at all and this stamp is then used to determine if the makefile needs re-construction. If the makefile is re-constructed during a call to make then a make clean operation will be carried out to ensure the entire project is re-built. This should ensure that the state of the project, makefile and the build remains in sync.

Finally, if your project needs to install the library and/or include files to a specific location, that behaviour is also catered for in BuildSys. Simply specify the install locations with the InstallLibraryName and InstallIncludeName arguments in the BSysProject initializer. For instance,

Project <- new("BSysProject", WorkingFolder="./MyFolder", Flat=F, InstallLibraryName="inst/lib", InstallIncludeName="inst/include")

Loading and Unloading the Library

Once the project is made using the make method the dynamic library can be loaded into R memory. We do so using the loadLibrary method and we can similarly unload the library using the unloadLibrary method. The library name is based upon the project name and the path to the library can be obtained with the libraryPath method. For example,

loadLibrary(Project) # load the library built by Project

unloadLibrary(Project) # unload the library built by Project

libraryPath(Project) # return the path to the library built by Project

Debugging the Library

To debug a dynamic library a debug build must be made to include the symbolic information needed with the dynamic library. By default, projects constructed using BuildSys will be created as debug builds. This is controlled by the Debug argument supplied in the call to the BSysProject initilizer. For instance,

Project <- new("BSysProject", Debug=TRUE)

creates a debug enabled project whereas,

Project <- new("BSysProject", Debug=FALSE)

create a release / optimised project. For an existing project we can make a debug build by supplying the Debug argument to the call to the make method as with,

Project <- make(Project, Debug=TRUE)

Recall that for S4 class methods to change the object state we need to assign the returned modified object to the original one, which is why the above example assigns the make call to Project.

Debugging is handled through Microsoft Visual Code and gdb / lldb and it is assumed that these components are installed and functional. For information regarding the installation of these components consult the next section.

To debug an existing dynamic library (built from a BSysProject) simply call the vcDebug method as follows.

vcDebug(Project) # Debug the library created by project

Calling this method will open an instance of Microsoft Visual Code that is correctly initialised to debug your library code. Within Visual Code you can open the source file/s of your library, set breakpoints and run a debug session. To run a debug session select the run/Start Debugging menu item. Doing so will result in a new instance of R being launched which contains the same environment (including loaded packages and loaded dynamic libraries) as the parent R session where vcDebug was first called. This R session is your sandbox to safely debug your library in and leaves the parent R session safe from loss should your code crash R completely.

Typically when debugging a new library the library requires R setup code to initialise as a foundation for library use and/or testing. With BuildSys you should perform this initial setup of R in the parent R session before calling vcDebug so that you never have to initialise R in your debug session. Its correct state will have been initialised from the parent R session.

Software Installation Requirements for Debugging

In order to debug code with BuildSys Microsoft Visual Code must be installed. You can download and install this software from here:

Download Visual Studio Code

Choose the appropriate installer for your operating system. On Windows and Linux the installer should include code in the PATH environment variable. Verify this by opening a new shell/command prompt and typing,

code

and enter. If Visual Code starts then the PATH enviroment variable is correctly set. If it fails to start then add it to the system PATH environment variable.

On MacOS the application name is Visual Studio Code.app and is an application bundle. For it to be usable by BuildSys drag the Visual Studio Code application bundle to the Applications folder using Finder.

On Windows you will have to install Rtools which you can find here:

Using Rtools40 on Windows

You will need to add R and Rtools to the system Path. For instance add,

C:\rtools40\mingw64\bin\
C:\rtools40\usr\bin\
C:\Program Files\R\R-4.0.1\bin\x64\

but be aware that the locations may differ in your case depending on which version of R and Rtools and where these products were installed. To set environment variables in Windows 10,

  1. Open the Start Search, type in “env”, and choose “Edit the system environment variables”:

  2. Click the “Environment Variables...” button.

  3. Set the environment variables as needed. The New button adds an additional variable.

  4. Dismiss all of the dialogs by choosing “OK”. Your changes are saved!

In Windows after installing Rtools we also need to install gdb. To install gdb, open an MSYS2 shell (In explorer migrate to the Rtools40 folder and double click on msys2.exe) and enter the following commands,

pacman -Sy
pacman -S mingw-w64-{i686,x86_64}-gdb

After installing verify that gdb is installed by opening a new DOS box and typing,

gdb -v

If all is correct it should respond with something like,

GNU gdb (GDB) 8.3.1
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

On MacOS / OS X you will need to install Xcode to have access to both clang and lldb. Go to Mac App Store and download and install Xcode.

On Linux you will need to have gcc installed. Installation of gcc on Linux is omitted because of the many and varied ways to do so depending on the distribution being used. Please consult Google on this topic for more information on installing development tools on Linux.

On Windows and Linux, you will also need to install the Microsoft C/C++ IntelliSense, debugging, and code browsing extension in Microsoft Visual Code. To install, press Ctrl+Shift+X and search for C/C++ and then click on Install beside the Microsoft C/C++ IntelliSense, debugging, and code browsing entry in the list. On MacOS / OS X you will need to install the Microsoft C/C++ IntelliSense, debugging, and code browsing and CodeLLDB extensions. To install press CMD+Shift+X and search for C/C++ and then click on Install beside the Microsoft C/C++ IntelliSense, debugging, and code browsing entry in the list, then also search for CodeLLDB and click on the Install next to Native debugger based on LLDB in the list.

On MacOS (OS X 10.11 and later) Apple introduced System Integrity Protection which restricts the ability to debug your code from within an R session. To debug code in R using BuildSys and CodeLLDB will require that the debugging restrictions are disabled. To do so shutdown MacOS, then press and hold down CMD+R keys while pressing the power button and keep them depressed until you see a language selection dialog. After selecting the language you should see the Utilities Window. In the menu bar at the top of the screen select the utilities/Terminal menu and in the terminal session enter the follow command:

csrutil enable --without debug

You will be prompted for your password. Enter your password to carry out the command. If successful you will be presented with a warning about it being a non-standard configuration. Now re-boot your Mac by selecting the re-boot option from the Apple menu at the top of the screen. You should now be able to debug your R dynamic libraries with BuildSys.

Note

If the absolute path to source and dynamic library files contains spaces then debugging in Visual Code fails to function correctly. To ensure code is debuggable users will need to avoid putting code in folders with whitespace in the folder names.

Author(s)

Paavo Jumppanen [aut, cre]

Maintainer: Paavo Jumppanen <paavo.jumppanen@csiro.au>

See Also

Debug C++ in Visual Studio Code Disabling and Enabling System Integrity Protection Enabling parts of System Integrity Protection while disabling specific parts? make buildMakefile vcDebug loadLibrary unloadLibrary libraryPath sourcePath includePath objPath installLibraryPath installIncludePath clean

Examples

ProjectFolder <- tempdir()

# Create source file for finite convolution example in "Writing R Extensions"  
lines <- c(
"#include <R.h>",
"#include <Rinternals.h>",
"",
"SEXP convolve2(SEXP a, SEXP b)",
"{",
"    int na, nb, nab;",
"    double *xa, *xb, *xab;",
"    SEXP ab;",
"",
"    a = PROTECT(coerceVector(a, REALSXP));",
"    b = PROTECT(coerceVector(b, REALSXP));",
"    na = length(a); nb = length(b); nab = na + nb - 1;",
"    ab = PROTECT(allocVector(REALSXP, nab));",
"    xa = REAL(a); xb = REAL(b); xab = REAL(ab);",
"    for(int i = 0; i < nab; i++) xab[i] = 0.0;",
"    for(int i = 0; i < na; i++)",
"        for(int j = 0; j < nb; j++) xab[i + j] += xa[i] * xb[j];",
"    UNPROTECT(3);",
"    return ab;",
"}")  

SourceFilePath <- paste0(ProjectFolder, "/convolve.c")
writeLines(lines, SourceFilePath)

# digest need not be loaded but the digest package needs to be installed
# as it is used to create a digest of the project to track the need for
# makefile re-creation.
require(BuildSys)

# create project to build shared library, a flat project with source in current working directory. 
Project <- new("BSysProject", ProjectFolder)

# re-initialise project from current working directory, new("BSysProject") calls this internally
Project <- initProjectFromFolder(Project, ProjectFolder)

# build the shared library
make(Project)

# get project library path
libraryPath(Project)

# get project source path
sourcePath(Project)

# get project include path
includePath(Project)

# get project object path
objPath(Project)

# get project install library path
installLibraryPath(Project)

# get project install include path
installIncludePath(Project)

# load the library
loadLibrary(Project)

# R wrapper on .Call
conv <- function(a, b) .Call("convolve2", a, b)

# Test data
a <- rnorm(100)
b <- rep(1.0, times=10)

# call the shared library function
conv(a, b)

## Not run: 
# open a debug session - assumes Visual Studio Code is installed as directed in the 
# package documentation. This will open a Visual Studio Code session. In that session
# open the convolve.c source file and set breakpoints, then select the "run/start debugging"
# menu to debug. This will start a new R session with the same state as the parent 
# R session where vcDebug() is first called. In the new debug session run the following
# command:
#
# conv(a, b)
vcDebug(Project)

## End(Not run)

# unload the shared library
unloadLibrary(Project)

# clean up example
make(Project, "clean")
clean(Project)
unlink(SourceFilePath)

[Package BuildSys version 1.1.2 Index]