MuzHack Has a Forum

I have just started a forum for MuzHack, based on the amazing Discourse. Stay tuned for useful content.

MuzHack Is in Production

For some months now I have been developing MuzHack, a platform for makers of music hardware to publish their designs. I’m pleased to say it’s finally in production! The project is in collaboration with the Norwegian music sound research center Notam. My hope is it will spark inspiration between independent hardware developers, similar to how I and so many other software developers have benefitted from GitHub.

Stay tuned for more news!

Building ChucK(JS) with Gradle

As part of my work to port the ChucK music programming language to JavaScript, via the Emscripten C++ to JavaScript compiler, I’ve implemented a Gradle build script for it (instead of the original Makefile). Aside from my working for Gradle, I think it’s a great choice for this kind of polyglot project, where both native and JavaScript targets are built. ChucK is also highly cross platform (i.e. Linux, OS X, Windows), which makes writing a general build script for it demanding. Gradle is a huge help here, especially after its recently gaining support for C and C++.

Aside from the build script itself, ‘build.gradle’, I had to include a wrapper script (generated by my global installation of Gradle) ‘gradlew’ (‘gradlew.bat’ for Windows), as well as ‘gradle/wrapper/gradle-wrapper.jar’ and ‘gradle/wrapper/gradle-wrapper.properties’.

Configurability

The build has been made user configurable along two axes: Debuggability and Linux sound architecture. The first means that unless -P debug=false is specified on the command line, targets are built for debuggability. Otherwise, targets will be built in optimized mode. The latter means that on Linux, one can choose between three sound architectures: ALSA (-P audioArchitecture=alsa, the default), PulseAudio (-P audioArchitecture=pulse) and JACK (-P audioArchitecture=jack).

Modularization

The Gradle build script has been modularized into plugins for the Lex/Yacc and Emscripten tasks, in order to keep the main script as declarative as possible. Furthermore, the Emscripten plugin is only loaded if Emscripten is detected in the OS environment.

Native Compilation

The native (i.e. non-JavaScript) version of ChucK is simply a console executable, which makes it fairly straightforward to build via Gradle’s component model. Such a target is simply represented by a NativeExecutableSpec in the component model, so for the most part the difficulty lies in determining platform dependent parameters and which C and C++ files to compile. Lex and Yacc tools must also be invoked in order to compile the lexer and compiler components respectively, for which there must be custom build steps (due to lack of direct support in the component model).

To build the ChucK console command (‘build/binaries/chuckExecutable/chuck’), simply issue the following command: ./gradlew assemble.

The main build script (build.gradle) looks as follows:

wrapper {
  gradleVersion = "2.5"
}

apply plugin: 'cpp'
apply plugin: 'c'
apply plugin: 'lexAndYacc'

enum Platform {
  OsX('mac os x|darwin|osx'),
  Windows('windows'),
  Linux('linux')
  Platform(String regex) { this.regex = regex }
  String regex
  static Platform current() {
    def osName = System.getProperty('os.name')
     values().find { osName =~ "(?i)${it.regex}" }
   }
}

ext.platform = Platform.current()
def debug = true
if (project.hasProperty('debug')) {
  debug = project.debug.toLowerCase().matches(/^(true|1)$/)
}
if (debug) {
  if (project.platform == Platform.OsX || project.platform == Platform.Linux) {
    ext.compilerFlags = ['-g',]
  }
} else {
  if (project.platform == Platform.OsX || project.platform == Platform.Linux) {
    ext.compilerFlags = ['-O3',]
  }
}

binaries.all {
  // Define toolchain-specific compiler and linker options
  cCompiler.args project.compilerFlags as String[]
  cppCompiler.args project.compilerFlags as String[]

  if (project.platform == Platform.OsX) {
    // TODO: Determine automatically
    cCompiler.args '-DHAVE_CONFIG_H', '-D__MACOSX_CORE__', '-isysroot',
      '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk', \
      '-mmacosx-version-min=10.4'
    cppCompiler.args '-D__MACOSX_CORE__', '-isysroot', \
      '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk', \
      '-mmacosx-version-min=10.4'
    linker.args '-F/System/Library/PrivateFrameworks', '-weak_framework', 'MultitouchSupport', \
      '-framework', 'CoreAudio', '-framework', 'CoreMIDI', '-framework', 'CoreFoundation', '-framework', \
      'IOKit', '-framework', 'Carbon', '-framework', 'AppKit', '-framework', 'Foundation', \
      '-mmacosx-version-min=10.4'
  } else if (project.platform == Platform.Linux) {
    if (!debug) {
      cCompiler.args '-fno-strict-aliasing'
      cppCompiler.args '-fno-strict-aliasing'
    }
    cCompiler.args '-DHAVE_CONFIG_H', '-D__PLATFORM_LINUX__ -D__CK_SNDFILE_NATIVE__'
    cppCompiler.args '-D__PLATFORM_LINUX__ -D__CK_SNDFILE_NATIVE__'

    def audioArch = (project.hasProperty('audioArchitecture')) ? audioArchitecture.toLowerCase() : 'alsa'
    if (audioArch == 'alsa') {
      cCompiler.args '-D__LINUX_ALSA__'
      cppCompiler.args '-D__LINUX_ALSA__'
      linker.args '-lasound'
    } else if (audioArch == 'jack') {
      cCompiler.args '-D__UNIX_JACK__'
      cppCompiler.args '-D__UNIX_JACK__'
      linker.args '-lasound', '-ljack '
    } else if (audioArch == 'pulse') {
      cCompiler.args '-D__LINUX_PULSE__'
      cppCompiler.args '-D__LINUX_PULSE__'
      linker.args '-lasound', '-lpulse-simple', '-lpulse'
    } else {
      throw new Exception("Unknown audioArchitecture option '${audioArchitecture}'")
    }

    linker.args '-lstdc++', '-ldl', '-lm', '-lsndfile', '-lpthread'
  }
}

model {
  components {
    chuck(NativeExecutableSpec) {
      sources.all {
        exportedHeaders {
          srcDir 'src'
          srcDir 'src/lo'
        }
      }
      sources {
        yaccOutput(CSourceSet) {
          generatedBy tasks.yacc
          source.include '*.c'
        }
        lexOutput(CSourceSet) {
          generatedBy tasks.lex
          source.include '*.c'
          lib sources.yaccOutput
        }
        c {
          source {
            srcDir 'src'
            include '*.c'
            include 'lo/*.c'
            if (project.platform != Platform.Windows) {
              exclude 'chuck_win32.c'
            }
            if (project.platform != Platform.OsX) {
              exclude 'util_sndfile.c'
            }
          }
          lib sources.lexOutput
          lib sources.yaccOutput
        }
        cpp {
          source {
            srcDir 'src'
            include '*.cpp'
            include 'RtAudio/*.cpp'
            exclude 'rtaudio_c.cpp'
            exclude 'chuck_js.cpp'
            exclude 'digiio_webaudio.cpp'
          }
        }
      }
    }
  }
}

if (System.getenv("EMSCRIPTEN")?.trim()) {
  logger.info("Emscripten detected - Configuring targets")
  apply plugin: 'emscripten'
} else {
  logger.info("Emscripten not detected - Not configuring targets")
}

Lex and Yacc Plugin

The tasks for compiling the lexer and compiler components are modularized into a separate plugin, ‘buildSrc/src/main/groovy/LexAndYaccPlugin.groovy’. To make the plugin importable by the build script, it has an accompanying metadata file ‘buildSrc/src/main/resources/META-INF/gradle-plugins/lexAndYacc.properties’, with the following content:

implementation-class=LexAndYaccPlugin

The plugin itself is implemented in the following way:

import org.gradle.api.*
import org.gradle.api.file.*
import org.gradle.api.tasks.*

class LexCompiler extends DefaultTask {
  @InputFiles FileCollection lexFiles
  @OutputDirectory File sourceDir
  @OutputDirectory File headerDir

  @TaskAction
  void processLexFiles() {
    lexFiles.each { file ->
      project.exec {
        commandLine = ['flex', '-o', new File(sourceDir, file.name - '.lex' + '.yy.c').absolutePath,
          file.absolutePath]
      }
    }
  }
}

class YaccCompiler extends DefaultTask {
  @InputFiles FileCollection yaccFiles
  @OutputDirectory File sourceDir
  @OutputDirectory File headerDir

  @TaskAction
  void processYaccFiles() {
    yaccFiles.each { file ->
      project.exec {
        commandLine = ['bison', '-dv', '-o', new File(sourceDir, file.name - '.y' + '.tab.c'),
          file.absolutePath]
      }
    }
  }
}

class LexAndYaccPlugin implements Plugin {
  void apply(Project project) {
    project.task('yacc', type: YaccCompiler) {
      sourceDir = project.file("${project.buildDir}/src/generated/yacc")
      headerDir = project.file("${project.buildDir}/src/generated/yacc")
      yaccFiles = project.files('src/chuck.y')
    }

    project.task('lex', type: LexCompiler) {
      sourceDir = project.file("${project.buildDir}/src/generated/lex")
      headerDir = project.file("${project.buildDir}/src/generated/lex")
      lexFiles = project.files('src/chuck.lex')
    }
  }
}

Building JavaScript Library via Emscripten

Here comes the interesting part (IMHO). To turn ChucK into a JavaScript library which can be run right inside your Web browser, we employ the wizardry of the Emscripten C++ to JavaScript compiler. This means that for 99% of the code, there’s a straight translation from the C/C++ source code to the JavaScript equivalent. Only a touch of adapter code is required to make ChucK interface with the Web Audio API.

Writing the build logic for the JavaScript target is a bit more challenging though, mainly because there is no direct support for Emscripten in Gradle. Instead, you have to write several tasks and task types, in order to compile the various sources through Emscripten and finally to link them together into a JavaScript library (chuck.js).

To compile the ChucK JavaScript library (‘build/js/chuck.js’), first activate the Emscripten SDK in your shell (would be sweet if Gradle provisioned it for you wouldn’t it?), and then issue the command ./gradlew emscripten.

As described earlier, the Emscripten part of the build logic is implemented as a separate plugin. It is made importable through a metadata file, ‘buildSrc/src/main/resources/META-INF/gradle-plugins/emscripten.properties’, with the following content:

implementation-class=EmscriptenPlugin

The Emscripten plugin itself, ‘buildSrc/src/main/groovy/EmscriptenPlugin.groovy’, is implemented as follows:

import org.gradle.api.*
import org.gradle.api.plugins.*
import org.gradle.api.file.*
import org.gradle.api.tasks.*
import org.gradle.api.tasks.incremental.IncrementalTaskInputs

class EmscriptenBaseTask extends DefaultTask {
  def emccPath = new File(System.getenv("EMSCRIPTEN"), 'emcc').absolutePath
}

class EmscriptenCompiler extends EmscriptenBaseTask {
  @InputFiles FileCollection emscriptenFiles
  @OutputDirectory File outputDirectory

  def replaceExtension(fileName) {
    return fileName.replaceFirst(/\.(c|cpp)$/, '.o')
  }

  @TaskAction
  void processEmscriptenFiles(IncrementalTaskInputs inputs) {
    inputs.outOfDate { change ->
      project.exec {
        // TODO: Don't hardcode YACC header dir
        commandLine = [emccPath, '-Isrc', '-Isrc/lo', '-Ibuild/src/generated/yacc', '-o',
            new File(outputDirectory, replaceExtension(change.file.name)).absolutePath] +
        project.compilerFlags + [change.file.absolutePath]
      }
    }

    inputs.removed { change ->
      new File(outputDirectory, replaceExtension(change.file.name)).delete()
    }
  }
}

class EmscriptenLinker extends EmscriptenBaseTask {
  @InputFiles FileCollection emscriptenFiles
  @OutputDirectory File outputDirectory

  @TaskAction
  void processEmscriptenFiles() {
    project.exec {
      // TODO: Make -g and SAFE_HEAP configurable
      commandLine = [emccPath, '-g', '-s', 'EXPORTED_FUNCTIONS=["_executeCode"]', '--js-library',
        'src/emscripten/libraries/webaudio.js', '-s', 'SAFE_HEAP=1', '-s', 'DEMANGLE_SUPPORT=1', '-o',
        new File(outputDirectory, 'chuck.js').absolutePath] + emscriptenFiles.collect { it.absolutePath }
    }
  }
}

class EmscriptenPlugin implements Plugin {
  void apply(Project project) {
    project.task('emscriptenYacc', type: EmscriptenCompiler) {
      outputDirectory = project.file("${project.buildDir}/emscripten")
      emscriptenFiles = project.fileTree(project.tasks.yacc.sourceDir)
        .include('chuck.tab.c')
      dependsOn(project.tasks.yacc)
    }

    project.task('emscriptenLex', type: EmscriptenCompiler) {
      outputDirectory = project.file("${project.buildDir}/emscripten")
      emscriptenFiles = project.fileTree(project.tasks.lex.sourceDir)
        .include('chuck.yy.c')
      dependsOn(project.tasks.lex)
      dependsOn(project.tasks.yacc)
    }

    project.task('emscriptenCompile', type: EmscriptenCompiler) {
      outputDirectory = project.file("${project.buildDir}/emscripten")
      emscriptenFiles = project.fileTree(dir: 'src')
        .include('*.c')
        .include('*.cpp')
        .exclude('rtaudio_c.cpp')
        .exclude('digiio_rtaudio.cpp')
        .exclude('util_sndfile.c')
        .exclude('chuck_win32.c')

      dependsOn(project.tasks.yacc)
    }

    project.task('emscripten', type: EmscriptenLinker) {
      outputDirectory = project.file("${project.buildDir}/js")
      emscriptenFiles = project.tasks.emscriptenYacc.outputs.files.asFileTree + \
        project.tasks.emscriptenLex.outputs.files.asFileTree + \
        project.tasks.emscriptenCompile.outputs.files.asFileTree
    }
  }
}

To see the whole build script, go here.

In a future blog post, I will detail how to build also the ChucKJS example HTML pages with Gradle, replacing Grunt in the process.

How to Access Real DOM Nodes with Mercury/Virtual-DOM, Part 2

In the time since my first post on accessing non-virtual DOM nodes with Mercury/virtual-dom, I’ve learnt that my approach wasn’t as sound as I initially thought. The creator of Mercury, Jake Verbaten, let me know that virtual-dom hooks should only modify node attributes, otherwise the behaviour will be undefined (“innerHTML in hooks breaks the tree indexing that virtual-dom is using”).

Instead, I learnt, one should use Mercury Widgets to render directly to the DOM. These aren’t subject to the same restrictions, so it turns out I should’ve been writing widgets instead of hooks.

Armed with this new knowledge, I wrote an example on accessing the non-virtual (real) DOM with either hooks (for manipulating node attributes) or widgets (for rendering new nodes), which was integrated into the Mercury project as real-dom.js. I’ve posted an extract of the example below, which shows only how to render a DOM node in a Widget:

'use strict';
var hg = require('mercury');

function Widget() {
    // This is required for an instance to be recognized by Mercury as a widget
    this.type = 'Widget';
}

Widget.prototype.init = function init() {
    var elem = document.createElement('div');
    elem.innerHTML = 'Content set directly on real DOM node, by widget ' +
        '<em>before</em> update.';
    return elem;
};
Widget.prototype.update = function update(prev, elem) {
    elem.innerHTML = 'Content set directly on real DOM node, by widget ' +
        '<em>after</em> update.';
};

function App() {
    var state = hg.state({
        updated: hg.value(false)
    });
    setTimeout(function timer() {
        state.updated.set(true);
    }, 2000);
    return state;
}

App.render = function App(state) {
    // Create widget controlled node
    return new Widget();
};

hg.app(document.body, App(), App.render);

Uploading Resized Pictures to S3 in Meteor Application

I am facing the need to upload pictures to Amazon S3 in my Meteor application, MuzHack. The pictures must be resized before storing in S3, and there’s the option of doing this either on my server or in the client. I have elected to go with the latter for now, since it’s simple and doesn’t put load on my server.

To resize pictures client side I am using the Meteor Clientside Image Manipulation library, and to upload pictures to S3 I am using Slingshot.

Combining Meteor Clientside Image Manipulation and Slingshot

Combining the two libraries isn’t straightforward. Meteor Clientside Image Manipulation returns a data URI, while Slingshot requires a file or a blob.

The data URI format returned by Meteor Clientside Image Manipulation looks like data:$type;base64,$content. The type and the content can be used to create a blob, for compatibility with Slingshot. The following code (CoffeeScript) shows how I did this:

Example Code for Combining Meteor Clientside Image Manipulation and Slingshot

b64ToBlob = (b64Data, contentType, sliceSize) ->
  sliceSize = sliceSize || 512

  byteCharacters = atob(b64Data)
  byteArrays = []
  offset = 0
  while offset < byteCharacters.length
    slice = byteCharacters.slice(offset, offset + sliceSize)
    byteNumbers = (slice.charCodeAt(i) for i in [0...slice.length])
    byteArray = new Uint8Array(byteNumbers)
    byteArrays.push(byteArray)
    offset += sliceSize

uploader = new Slingshot.Upload("pictures")
new Promise((resolve, reject) ->
  processImage(file, 300, 300, resolve)
)
  .then((dataUri) ->
    match = /^data:([^;]+);base64,(.+)$/.exec(dataUri)
    [file.name, match[1], match[2]]
  )
  .then(([name, type, b64]) ->
    new Promise((resolve, reject) ->
      blob = b64ToBlob(b64, type)
      blob.name = name
      uploader.send(blob, (error, downloadUrl) ->
        if error?
          reject(error.message)
        else
          resolve(downloadUrl)
      )
    )
  )

How to Access Real DOM Nodes with Mercury/virtual-dom

At work, we are using the Mercury JavaScript UI library, which is a more functional alternative to React (at least this is my understanding of it). It is also highly modular, in that it’s simply composed from several independent libraries (such as virtual-dom).

Like React, Mercury hides the browser DOM behind a virtual model, to improve performance. However, this can cause problems in case you need real access to the DOM. For example, a 3rd party library such as the popular spin.js may require changing the DOM directly, and consequently, the virtual DOM will not suffice.

There is a way to get hold of the real DOM node corresponding to a virtual node thankfully, but it isn’t straightforward.

Virtual Node Hooks

The virtual-dom library supports associating virtual nodes with hooks to be called when a corresponding DOM node is rendered and when it’s removed, respesctively. This is done by supplying an object to the virtual node constructor which has in its prototype a ‘hook’ method and/or an ‘unhook’ method. The former gets called when the DOM node has been rendered and the latter after it has been removed. Both methods receive the DOM node as their first argument.

Below is a full example that shows how to register the aforementioned ‘hook’ and ‘unhook’ callbacks.

'use strict';
var hg = require('mercury');
var h = require('mercury').h;

function Hook() {}
// Hook to be called when DOM node has been rendered
Hook.prototype.hook = function hook(node) {
    node.innerText = 'This is set directly on the real DOM node.';
};
// Hook to be called when DOM node is removed
Hook.prototype.unhook = function unhook(node) {
    node.innerText = 'Unhook hook called';
};

function App() {
    var state = hg.state({
        remove: hg.value(false)
    });
    setTimeout(function timer() {
        state.remove.set(true);
    }, 2000);
    return state;
}

App.render = function App(state) {
    return !state.remove ?
        // Create virtual node with hook
        h('div', {
            hook: new Hook()
        }) :
        // Node without hook
        h('div');
};

hg.app(document.body, App(), App.render);

Continuous Deployment of MuzHack with Docker on Tutum

I have recently implemented continuous deployment of my MuzHack Web application to the Docker hosting service Tutum, in both production and staging versions. I am really pleased with the smoothness of the whole process of setting it up, and the ease of day-to-day administration.

Docker Image Definition

The very first thing I had to do was to define MuzHack’s Docker image. This is done in the so-called Dockerfile. I base it off the standard Node image. Then I add the PhantomJS headless browser, since it’s required by Meteor (for SEO purposes), and convert MuzHack from a Meteor application into a Node equivalent.

Docker Image Registry

Before being able to deploy, I needed Docker image hosting. I decided to use the service Quay.io, which is free for open source projects, and supports automatically building images when new commits are pushed to corresponding GitHub repositories. Importantly, Quay can also notify other services after building an image.

Setting Up GitHub Trigger

In order to trigger Quay upon pushing to the master branch of the MuzHack repository, I had to create a “Custom Git Repository Push” trigger in the Quay project’s settings (I couldn’t quite make the standard “GitHub Repository Push” trigger work). This required installing a corresponding webhook in the settings of the GitHub repository as well. Once this was done, continuous building of images hosted on Quay worked beautifully.

Setting up Tutum Trigger

In order to implement the next part of the chain, continuous deployment, I had to add a “Webhook POST” notification for the “Push to Repository” event in Quay. This is configured to invoke a trigger on Tutum, which re-deploys the staging version of the MuzHack service. But more on this later.

Deployment to Docker Hosting (Tutum)

Having solved the problem of hosting the Docker images, I could move on to setting up deployment of MuzHack to a Docker hosting service. I decided on Tutum, since I found it offered the best available experience due to its Stack model, which makes it a breeze to maintain different environments (e.g. staging and production). It also runs on top of the server provider DigitalOcean, which is already my preference in this space.

Stacks

To deploy to Tutum, I wrote a couple of stack definitions, that define three different Docker services per deployment (i.e. staging and production): MongoDB, MuzHack and Nginx (services correspond to Docker containers, although a service might spin up several containers to scale out). MongoDB is the database used by MuzHack and Nginx acts as a facade Web server, for basal duties such as handling SSL. The stack definitions ensure that MuzHack is linked to the MongoDB service and that Nginx is linked to MuzHack (as Nginx’ backend server).

Stacks are defined in YAML, and there is at the time of writing one for staging, tutum-staging.yaml, and one for production, tutum-production.yaml.

To create the stacks on Tutum, I used the tutum stack up command, e.g. for staging: tutum stack up -n tutum-staging -f docker/tutum-staging.yaml.

Automatic Deployment

So far, I’ve only set up automatic re-deployment of the staging version. This is done through a Tutum trigger for the MuzHack service of the Staging stack, which gets invoked by Quay.io when new images of the master branch are built. It works great by the way!

Administration of Deployments

Day to day administration of the MuzHack deployments is handled with ease through Tutum’s Web UI, which while relatively simplistic is really quite smart. After clicking one of the stacks (e.g. Staging), I can see its individual services and also the individual containers per service (even though I haven’t gone beyond one container per service yet).

So far, there hasn’t really been anything to administration duties, it just works 🙂 I can see the logs of the individual services, and there is basic monitoring. I would like better monitoring eventually (especially log searching), but unless I’m mistaken Tutum/Docker can be integrated with dedicated services for this.