Configuring YouCompleteMe for embedded software development

If you are a heavy vim user, you know no other editor (excepting derivatives like neovim) can beat the code editing speed you get once you develop the required muscle memory. But the standard vim install lacks the fancy development features available at most heavyweight IDEs: fuzzy code completion, refactoring, instant error highlighting, etc. Here is where YouCompleteMe (YCM) comes to rescue, adding all these features to vim, while preserving everything you like from everyone's favorite text editor.

There are a lot of guides on the net detailing how to install, configure and use YCM, including the official ones, so I will refrain from repeating them. But after a bit of searching, I could not find a tutorial explaining how to configure YCM for embedded software development in C/C++ language. It is not very difficult, but there are some tricky steps that can be hard to guess, so I decided to write down a small guide here.

In this entry I will explain how to create a .ycm_extra_conf.py file and tune it to develop for a specific hardware platform. As an example I will show the modifications required to develop for Espressif ESP32 microcontrollers using their official SDK esp-idf.

Prerequisites

To follow this tutorial you will need the following:

• A working vim + YouCompleteMe setup.
  
• A working cross compiler install + SDK for the architecture of your choice. In this blog entry I will be using esp-idf v4.2 for Espressif ESP32 microcontrollers. This includes a cross GCC setup for the Tensilica Xtensa CPU architecture.

The base system used in this guide is a Linux machine. I suppose you can follow this guide for macos and Windows machines just by changing the paths, but I have not tested it.

Procedure

Configuring an embedded development project for YCM to work properly requires the following steps:

• Creating the base .ycm_extra_conf.py configuration file.
  
• Configuring the toolchain and SDK paths.
• Obtaining the target compilation flags and adding them to the config file.
• Removing unneeded and conflicting entries from the compilation flags.
• Adding missing header paths and compile flags.
  

The configuration file

The usual way you configure YCM, is by having a .ycm_extra_conf.py file sitting in the root directory of every one of your C/C++ projects. Creating these files for non-cross platform development is usually easy and you can configure vim to use a default config file for all your projects (so you do not have to copy the .ycm_extra_conf.py to every project). If your build system uses make, cmake, qmake or autotools, you have a chance to generate the file automatically using tools like YCM-Generator. But in my experience, for cross platform development environments using complex build systems (like OpenWRT, ninja, complex makefiles, etc.), you end up having to manually customize the configuration file yourself. And that's exactly what I will do in this guide, so let's get to work!

The base config

The first step is to grab somewhere a .ycm_extra_conf.py example configuration file. You can search elsewhere or use this one I trimmed from a bigger one obtained using YCM-Generator on a simple project:

from distutils.sysconfig import get_python_inc
import platform
import os.path as p
import os
import subprocess

DIR_OF_THIS_SCRIPT = p.abspath( p.dirname( __file__ ) )
DIR_OF_THIRD_PARTY = p.join( DIR_OF_THIS_SCRIPT, 'third_party' )
DIR_OF_WATCHDOG_DEPS = p.join( DIR_OF_THIRD_PARTY, 'watchdog_deps' )
SOURCE_EXTENSIONS = [ '.cpp', '.cxx', '.cc', '.c', '.m', '.mm' ]

database = None

# These are the compilation flags that will be used in case there's no
# compilation database set (by default, one is not set).
# CHANGE THIS LIST OF FLAGS. YES, THIS IS THE DROID YOU HAVE BEEN LOOKING FOR.
flags = [
]

# Set this to the absolute path to the folder (NOT the file!) containing the
# compile_commands.json file to use that instead of 'flags'. See here for
# more details: http://clang.llvm.org/docs/JSONCompilationDatabase.html
#
# You can get CMake to generate this file for you by adding:
#   set( CMAKE_EXPORT_COMPILE_COMMANDS 1 )
# to your CMakeLists.txt file.
#
# Most projects will NOT need to set this to anything; you can just change the
# 'flags' list of compilation flags. Notice that YCM itself uses that approach.
compilation_database_folder = ''


def IsHeaderFile( filename ):
  extension = p.splitext( filename )[ 1 ]
  return extension in [ '.h', '.hxx', '.hpp', '.hh' ]


def FindCorrespondingSourceFile( filename ):
  if IsHeaderFile( filename ):
    basename = p.splitext( filename )[ 0 ]
    for extension in SOURCE_EXTENSIONS:
      replacement_file = basename + extension
      if p.exists( replacement_file ):
        return replacement_file
  return filename


def PathToPythonUsedDuringBuild():
  try:
    filepath = p.join( DIR_OF_THIS_SCRIPT, 'PYTHON_USED_DURING_BUILDING' )
    with open( filepath ) as f:
      return f.read().strip()
  except OSError:
    return None


def Settings( **kwargs ):
  # Do NOT import ycm_core at module scope.
  import ycm_core

  global database
  if database is None and p.exists( compilation_database_folder ):
    database = ycm_core.CompilationDatabase( compilation_database_folder )

  language = kwargs[ 'language' ]

  if language == 'cfamily':
    # If the file is a header, try to find the corresponding source file and
    # retrieve its flags from the compilation database if using one. This is
    # necessary since compilation databases don't have entries for header files.
    # In addition, use this source file as the translation unit. This makes it
    # possible to jump from a declaration in the header file to its definition
    # in the corresponding source file.
    filename = FindCorrespondingSourceFile( kwargs[ 'filename' ] )

    if not database:
      return {
        'flags': flags,
        'include_paths_relative_to_dir': DIR_OF_THIS_SCRIPT,
        'override_filename': filename
      }

    compilation_info = database.GetCompilationInfoForFile( filename )
    if not compilation_info.compiler_flags_:
      return {}

    # Bear in mind that compilation_info.compiler_flags_ does NOT return a
    # python list, but a "list-like" StringVec object.
    final_flags = list( compilation_info.compiler_flags_ )

    # NOTE: This is just for YouCompleteMe; it's highly likely that your project
    # does NOT need to remove the stdlib flag. DO NOT USE THIS IN YOUR
    # ycm_extra_conf IF YOU'RE NOT 100% SURE YOU NEED IT.
    try:
      final_flags.remove( '-stdlib=libc++' )
    except ValueError:
      pass

    return {
      'flags': final_flags,
      'include_paths_relative_to_dir': compilation_info.compiler_working_dir_,
      'override_filename': filename
    }

  if language == 'python':
    return {
      'interpreter_path': PathToPythonUsedDuringBuild()
    }

  return {}


def PythonSysPath( **kwargs ):
  sys_path = kwargs[ 'sys_path' ]

  interpreter_path = kwargs[ 'interpreter_path' ]
  major_version = subprocess.check_output( [
    interpreter_path, '-c', 'import sys; print( sys.version_info[ 0 ] )' ]
  ).rstrip().decode( 'utf8' )

  sys_path[ 0:0 ] = [ p.join( DIR_OF_THIS_SCRIPT ),
                      p.join( DIR_OF_THIRD_PARTY, 'bottle' ),
                      p.join( DIR_OF_THIRD_PARTY, 'regex-build' ),
                      p.join( DIR_OF_THIRD_PARTY, 'frozendict' ),
                      p.join( DIR_OF_THIRD_PARTY, 'jedi_deps', 'jedi' ),
                      p.join( DIR_OF_THIRD_PARTY, 'jedi_deps', 'parso' ),
                      p.join( DIR_OF_THIRD_PARTY, 'requests_deps', 'requests' ),
                      p.join( DIR_OF_THIRD_PARTY, 'requests_deps',
                                                  'urllib3',
                                                  'src' ),
                      p.join( DIR_OF_THIRD_PARTY, 'requests_deps',
                                                  'chardet' ),
                      p.join( DIR_OF_THIRD_PARTY, 'requests_deps',
                                                  'certifi' ),
                      p.join( DIR_OF_THIRD_PARTY, 'requests_deps',
                                                  'idna' ),
                      p.join( DIR_OF_WATCHDOG_DEPS, 'watchdog', 'build', 'lib3' ),
                      p.join( DIR_OF_WATCHDOG_DEPS, 'pathtools' ),
                      p.join( DIR_OF_THIRD_PARTY, 'waitress' ) ]

  sys_path.append( p.join( DIR_OF_THIRD_PARTY, 'jedi_deps', 'numpydoc' ) )
  return sys_path

Pointing to headers

The base configuration file has everything necessary for YCM to work. Now we only need to tell it the build flags, including the header locations for the cross compiler and the SDK. So the first customization we do is defining two new variables with the paths to the cross compiler (GCC_PATH) and the toolchain (IDF_PATH). I am writing these in the config file (derived from HOME), but you could get them from environment variables. Note that the config file is a Python script, so you must use Python syntax for the variable definitions.

HOME = os.environ['HOME']
GCC_PATH = HOME + '/.espressif/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf'
IDF_PATH = HOME + '/dev/esp-idf'

Make sure these directories point to your compiler and environment locations and jump to the next step.

Getting the compile flags

Now we have to define the compile flags. We have to put them inside the flags array. Usually, the tricky part here is finding the compile flags for your project, especially with most modern build environments that tend to hide the compiler invocation command line during the build process. When using idf.py command (from esp-idf), it is as simple as supplying the -v switch. Some cmake projects require defining the VERBOSE=1 variable. When using OpenWRT build environment, you have to define the V=99 (or V=s) variable when you invoke make... There can be infinite ways to get the desired verbose output and you might have to find the one your build system uses. In the end you have to browse the verbose build output and spot a compiler invocation with all the parameters. Make sure the compiler invocation is to compile a file, and not to link a binary or create a library. In the end you should have something like this (a big chunk of the line has been redacted for clarity):

/home/esp/.espressif/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc -Iconfig -I../components/tarablessd1306 -I/home/esp/dev/esp-idf/components/newlib/platform_include -I/home/esp/dev/esp-idf/components/freertos/include -I/home/esp/dev/esp-idf/components/freertos/xtensa/include -I/home/esp/dev/esp-idf/components/heap/include -I/home/esp/dev/esp-idf/components/log/include -I/home/esp/dev/esp-idf/components/lwip/include/apps [...] -c ../components/tarablessd1306/ifaces/default_if_i2c.c

 Now you have the flags, but... you need them in Python array syntax. No problem, I made this tiny python script to do the conversion:

#!/usr/bin/env python3
import sys;

if __name__ == '__main__':
    for flag in sys.argv[1:]:
        if len(flag):
            print('    \'' + flag + '\',')

Write the script to a file, give it exec permission, and run it passing as arguments the complete line above. It will generate something like this:

    '/home/esp/.espressif/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc',
    '-Iconfig',
    '-I../components/tarablessd1306',
    '-I/home/esp/dev/esp-idf/components/newlib/platform_include',
    '-I/home/esp/dev/esp-idf/components/freertos/include',
    '-I/home/esp/dev/esp-idf/components/freertos/xtensa/include',
    '-I/home/esp/dev/esp-idf/components/heap/include',
    '-I/home/esp/dev/esp-idf/components/log/include',
    '-I/home/esp/dev/esp-idf/components/lwip/include/apps',
    [...]
    '-c',
    '../components/tarablessd1306/ifaces/default_if_i2c.c',

Now remove the first line that corresponds to the gcc binary, and copy all the others to .ycm_extra_conf.py file, inside the flags array.

Flags cleanup

We have copied all the flags, but we have to remove some of them because:

• Some flags are related to the input and output files, that will be supplied by YCM internals (e.g.: -c <param>, -o <param>).
  
• Some flags are GCC specific, but YCM uses clang for code analysis, so unless we remove them, we will get errors (e.g.: -mlongcalls, -fstrict-volatile-bitfields).
• Some flags are related to debug information or optimizations, and are thus not relevant for code analysis (e.g.: -ffunction-sections, -fdata-sections, -ggdb, -Og, -MD, -MT <param>, -MF <param>)
  
• Some flags are architecture specific and are most likely not supported by clang (e.g.: -mfix-esp32-psram-cache-issue, -mfix-esp32-psram-cache-strategy=memw)
• Some flags might cause sub-optimal error hinting. E.g. remove -Werror for the warnings to be properly highlighted as warnings instead of errors.
  

Another thing you can do is searching every path that matches the ESP_IDF variable definition, and replace it by this variable plus the remainder of the path. The idea is not having any absolute paths, to be able to adapt this script to other machines. When removing flags, if you have doubts, you can keep the flag, and remove it if YCM complains when you start vim later.

Cross compiler includes

By default, YCM will search your system header files (usually in /usr/include). This can work for many cases (especially if you are developing for a cross Linux platform) but will probably cause problems because your system header files are most likely partially incompatible with the ones from your cross compiler. To fix this problem, you have to tell clang not to use the standard include files, but to use the ones from your cross compiler. For the esp-idf compiler, you can achieve this by adding to the flags array:

    '-nostdinc',
    '-isystem', GCC_PATH + '/lib/gcc/xtensa-esp32-elf/8.4.0/include',
    '-isystem', GCC_PATH + '/lib/gcc/xtensa-esp32-elf/8.4.0/include-fixed',
    '-isystem', GCC_PATH + '/xtensa-esp32-elf/include',

The important detail here is using the -nostdinc flag to avoid the compiler using the system header files. Then supply the platform specific includes.

Additional tweaks

With this we should be ready to test our configuration. Make sure the config file is sitting in your project path and fire vim from there. Open a C/C++ file and YCM should be working, maybe perfectly, but maybe it will still highlight as errors or warnings some lines that are 100% OK. If you followed properly all the steps, and all the paths in the config file are correct, the remaining problems are most likely because we took compilation flags from a file, but there might be other files in the project that are built using different flags. What you will have to do here is browse the reported errors (:YcmDiags is useful here) and fix them by adding to the flags array the needed defines or include directories you lack. Depending on the number of flags you have to add here, this step might get a bit tedious, but each time you add a missing flag, you will be a step nearer to the YCM perfection. To get the esp-idf environment working properly, I had to manually add e.g. the following:

    '-DSSIZE_MAX=2147483647',
    '-I' + DIR_OF_THIS_SCRIPT + '/build/config',
    '-I' + IDF_PATH + '/components/nvs_flash/include',
    '-I' + IDF_PATH + '/components/spi_flash/include',
    '-I' + IDF_PATH + '/components/esp_http_client/include',
    '-I' + IDF_PATH + '/components/nghttp/port/include',
    '-I' + IDF_PATH + '/components/json/cJSON',

And that's all. It can take a while having this perfectly configured, but once done, the productivity enhancement can be very high!

You can find here the complete example .ycm_extra_conf.py file. If you have read through here, I hope you find this guide useful.

Happy hacking!