Kinect 2.0 Face Frame Data with Other Frame Data

Do you have a project that you are using the Kinect 2.0 Face detection as well as one, or more, of the other feeds from the Kinect? Well I am, and I was having issues with obtaining all the Frames I wanted from the Kinect. Let’s start with a brief, highlevel, overview, I had a need to obtain the data relating to the Color Image, the Body Tracking, and the Face Tracking. Seems very straight forward until I realized that the Face Data was not included in the MultiSourceFrameReader class. That reader only provided me the Color and Body frame data. In order to get the Face data I needed to use a FaceFrameReader. Which required me to listen for the arrival of two frame events.

For example, I was doing something like this.

public MainWindowViewModel()
{
	_kinectSensor = KinectSensor.GetDefault();

	const FrameSourceTypes frameTypes = FrameSourceTypes.Color | FrameSourceTypes.Body;
	MultiSourceFrameReader kinectFrameReader = _kinectSensor.OpenMultiSourceFrameReader( frameTypes );

	kinectFrameReader.MultiSourceFrameArrived += OnMultiSourceFrameArrived;

	const FaceFrameFeatures faceFrameFeatures = 
		  FaceFrameFeatures.BoundingBoxInColorSpace
		| FaceFrameFeatures.RotationOrientation
		| FaceFrameFeatures.FaceEngagement;

	faceFrameSource = new FaceFrameSource( _kinectSensor, 0, faceFrameFeatures );
	faceFrameReader = faceFrameSources.OpenReader();
	faceFrameReader.FrameArrived += OnFaceFrameArrived;

	_kinectSensor.IsAvailableChanged += OnSensorAvailableChanged;
	_kinectSensor.Open();

}

private void OnMultiSourceFrameArrived( object sender, MultiSourceFrameArrivedEventArgs e )
{
	//Process Color and Body Frames
}

private void OnFaceFrameArrived( object sender, FaceFrameArrivedEventArgs e )
{
	//Process Face frame
}

In theory, this should not be a problem because the Kinect is firing off all frames at around 30 per second and, I assume, in unison. However, I was running into the issue that if my processing of the Face data took longer then the 30th of a second I had to process, the color or body data could be at a different point in the cycle. What I was seeing was images that appeared to be striped between two frames. Now I understand that this behavior could be linked of various issues that I am not going to dive into in this post. But what I had noticed was the more processing I tried to pack in to the Face Frame arrival handler, the more frequently I saw bad images. It is worth noting that my actual project will process all six of the faces that the Kinect can track, and when having to iterate through more then one face per frame, the bad, striped, images were occurring more then good images. This lead me to my conclusion (and my solution lead me to write this blog.)

I also did not like the above approach because it forced me to process frames in different places, and possibly on different cycles. So when something wasn’t working I had to determine which Frame was the offender, then go to that processing method. No bueno.

In trouble shooting the poor images, I had the thought “I just want the color and frame that the Face frame using.” Confused? I’ll try to explain. Basically, the Kinect Face tracking is using some conglomeration of the basic Kinect Feeds (Color, Depth, Body) to figure out what is a face, and the features of that face. I know this because if a body is not being tracked, a face is not being tracked. The depth is then use to track if the eyes are open or closed and other intricacies of the face. Anyways, back on track, I had a feeling that the Kinect Face Frame had, at least, some link back to the other frames that were used to determine the state of the face for that 30th of a second. That is when I stumbled upon FaceFrame.BodyFrameReference and FaceFrame.ColorFrameReference (FaceFrame.DepthFrameReference also exsits, it’s just not needed for my purposes). From those references you can get the respective frames.

After my epiphany my code turned into:

public MainWindowViewModel()
{
	_kinectSensor = KinectSensor.GetDefault();

	//Multiframe reader stuff was here.
	//It is now gone.

	const FaceFrameFeatures faceFrameFeatures = 
		  FaceFrameFeatures.BoundingBoxInColorSpace
		| FaceFrameFeatures.RotationOrientation
		| FaceFrameFeatures.FaceEngagement;

	faceFrameSource = new FaceFrameSource( _kinectSensor, 0, faceFrameFeatures );
	faceFrameReader = faceFrameSources.OpenReader();
	faceFrameReader.FrameArrived += OnFaceFrameArrived;

	_kinectSensor.IsAvailableChanged += OnSensorAvailableChanged;
	_kinectSensor.Open();

}

private void OnFaceFrameArrived( object sender, FaceFrameArrivedEventArgs e )
{
	...
	var bodyReference = faceFrame.BodyFrameReference;
	BodyFrame bodyFrame = bodyReference.AcquireFrame();

	var colorReference = faceFrame.ColorFrameReference;
	ColorFrame colorFrame = colorReference.AcquireFrame();
	//Process Face frame
	...
}

private void ProcessBodyAndColorFrames( BodyFrame bodyFrame, ColorFrame colorFrame )
{
	//Process Body and Color Frames
	...
}

I still have the processing methods split out similarly to how I had them in the first (bad) way, but with this approach I am a little more confident that the Color and Body Frames and I analyzing are the same ones used by the Face processor. I am also more free to split up the processing as I see fit. The color and body are really only lumped together because that is how I was doing it before. In the future they might be split up and done in parallel, who knows.

And with that, the occurrence of bad images appears to be greatly reduced. At least for now. We will see how long it lasts. I still get some bad frames, but I am at least closer to being able to completely blame the Kinect, or poor USB performance, or something other than me (which is my ultimate goal.)

Not a big thing, just something I could not readily find on the web.

Javascript Handlebars Templates With Custom Helpers

Last time, I talked about Handlebars with Partials and before that it was Basic Handlebars Templates, so now I think it is time to talk about custom helper methods and what they allow us to do. If you have used Handlebars and have used the notation “#each” you have used a helper. Handlebars also lets us create our own Helper Methods, so we can make #DoMyCustomWork. I should also note that any Helper that uses the ‘#’ notation, needs to have the closing ‘/’ notation. For example, the built in “{{#if}}” needs to be closed with a “{{/if}}”.

I am going to build on my project from the last post in which I was using sport team information.

This time, I want to display the NHL teams sorted on the major segmentation, or conference, and the minor segmentation, or division. Basically, NHL teams are in a Division, and each Division belongs to a Conference and I want my list to reflect that.

Since I want to sort the teams based on Conference and Division, I feel like I should look at the most general split, or by conference. To do this I need something that will look at my data and only act on the teams that I want at the moment. It would be logical to think that one option would be to iterate through all the teams and use the built in #if helper. And if you thought that, you are wrong.
The #if only tests for True or “Falsey” values. Yes, “Falsey”. The #if helper checks if the value is “null”, “undefined”, and yes even “false”. A quick example: lets say I have data for a sports league, but the data on individual teams is empty, I can for example “{{#if teams}}” in a template, which will allow me to show the team data if is is there. Next I can put “{{else}}”, and put an error message saying “Team data not provided.”

Enough of that tangent. I want a custom helper that gives me all the teams of a certain NHL Conference. To do this I feel like I should make my helper as generic as possible because this time I want conference, but next time it might be divisions, or team color, or team location. So I want my helper to accept my collection, the property I want to search, and the value that I want to match on. With that criteria, this is what I came up with.

        Handlebars.registerHelper("where", function (collection, property, value, options) {
            var subset = [];

            for (var teamIndex in collection) {
                var team = collection[teamIndex];

                if (team[property] !== undefined && team[property].toLowerCase() === value.toLowerCase()) {
                    subset.push(team);
                }
            }

            return options.fn(subset);
        });

I put this snippet in a “registerHelpers()” function that lives in an object dedicated to Handlebars. But lets dive into the specifics a bit. What I am doing it telling Handlebars that I want to register a helper named “where,” which is a function that takes in 4 parameters: collection, property, value, and options. The first three I explained before. The last one, options, seems like an anomaly. In fact, options is a handlebars construct and that allows me to pass data from the JavaScript back to the template. Once in the function it’s just a basic JS function to iterate through my collection and compare the values, saving off the ones I want. After that has completed I give my subset back to the template by returning the subset through “options.fn().” Again, that is just how handlebars works.

To call this, I simply put :

{{#where <collection> <propertyToSearchOn> <valueToMatch> }}
1
in my template. The stuff in <> are placeholders for the actual values.
For my structure the actual line to call my helper is:
1
{{#where teams "conference" "west" }}

I should also point out that using #where, returns an object and shifts my context. That means using "{{#each teams}}" like in my previous post will no longer work. The object I have when I get to that point is not the same object as I had before. Have no fear, because #where returns just the object I am looking for in an array, I can just change the line of code to be "{{#each this}}." What I am saying there is "for each object in the collection" do whatever is in the each block.

So when I modify my template to use the #where helper

<script id="sports-template" type="text/x-handlebars-template">
    <div id="sport-list">
        {{#each leagues}}
        <div id="league-accordion" class="panel-group">
            <div class="league-header">
                <h1>{{{name}}} ({{acronym}})</h1>
            </div>

            <h3>Founded {{{founded}}}</h3>
            <div class="panel panel-default">
                {{#where teams "conference" "west" }}
                {{#each this}}
                <div class="conferenceDiv">
                    {{> teamTemplate}}
                </div>
                {{/each}}
                {{/where}}
            </div>
        </div>
        {{/each}}
    </div>
</script>

and run it, I get a list of the teams that have "West" listed as their conference.

The eagle eyed may have noticed that now I only have half my data. I am only showing teams in the West. How can I show teams for both Conferences? Well, I could simply duplicate my #where block and change the string to "east," but what is the fun in that, and I would be duplicating code. What I could do instead is look at my data then iterate the proper number of times.

I will do my best to make this more brief. I want to look at my data see how many Conferences I have in the NHL. Something like a Distinct call to a database. So I made a helper I call, "Distinct." It looks like this:

        Handlebars.registerHelper("distinct", function (listOfObjects, property, options) {
            var unique = {};
            var distinctList = [];

            for (var objectIndex in listOfObjects) {
                var object = listOfObjects[objectIndex];

                if (typeof (unique[object[property]]) == "undefined") {
                    distinctList.push(object[property]);
                }

                unique[object[property]] = 0;
            }

            return options.fn(distinctList);
        });

I call it like this:

{{#distinct teams "conference"}}

Then my full template turns into this:

<script id="sports-template" type="text/x-handlebars-template">
    <div id="sport-list">
        {{#each leagues}}
        <div id="league-accordion" class="panel-group">
            <div class="league-header">
                <h1>{{{name}}} ({{acronym}})</h1>
            </div>

            <h3>Founded {{{founded}}}</h3>

            {{#distinct teams "conference"}}
            {{#each this}}
            <div class="conferenceDiv">
                <h2>{{this}}</h2>
            </div>
            {{#where ../../teams "conference" this }}
            {{#each this}}
            <div class="conferenceDiv">
                {{> teamTemplate}}
            </div>
            {{/each}}
            {{/where}}

            {{/each}}
            {{/distinct}}
        </div>
        {{/each}}
    </div>
</script>

And now I get both the NHL conferences.

To get to the division level, I do a very similar pattern.
-Get the distinct divisions
-Grab the teams from those division
-Plop them in the DOM

If you want to see this, grab my project.

That explains helpers for Handlebars.

My project (again): Handlebars Sample

JavaScript Handlebars Templates with Partials

Intro
In a previous blog post I provided the basics for creating an HTML template using the handlebars JavaScript library. In that post, I stuck to a very basic template. This time I will kick it up a notch and cover partials. You might be asking “What do I need partials for?” and you asking gives me a way to start explaining. So thanks for asking.

Let’s say you are making several templates from some data that is similar, but not identical. For example, sports leagues. Sports leagues are very similar in how they are composed, but not always identical. Some leagues call the major split conferences others call them leagues. Inside of a league there might be divisions; who knows. But at some point, we get to a level where it’s a single team, and we want all teams to have the same visual representation. This is where the handlebars partial comes in to play. I can make a team template, and use it any other template I want. All I have to do is register that partial with the handlebars library, then reference the partial in my template, and BOOM!, I have the same look across all sports leagues, conferences, divisions, whatever.

For now, I am focusing on one league (the NHL), so each team object has the name of the team, the location, the conference, and the division. I want each team to have the same look, so I am going to make a team template.

<script id="team-template" type="text/x-handlebars-template">
    <div class="teamDiv">
        <h3>
            {{location}} {{name}}
        </h3>
    </div>
</script>

Yes, that template is pretty basic and, for the moment, could go in line with the rest of my template, but if later on I make the template more complex, or I change to have the teams separated out by the conference/division, or I add more sports, I might end up with duplicated code.

Now that I have made my partial, it needs to be registered before it can be used. That is easy to do, and looks like this:

Handlebars.registerPartial("teamTemplate", $("#team-template").html());

Finally, once I have told the Handlebars engine about the template, I can use that partial in other templates. That notation looks like this:

{{> teamTemplate}}

Anywhere that appears in a handlebars script, my partial will be used.

My full template looks like so:

<script id="sports-template" type="text/x-handlebars-template">
    <div id="sport-list">
        {{#each leagues}}
        <div id="league-accordion" class="panel-group">
            <div class="league-header">
                <h1>{{{name}}} ({{acronym}})</h1>
            </div>

            <h3>Founded {{{founded}}}</h3>
            <div class="panel panel-default">
                {{#each teams}}
                {{> teamTemplate}}
                {{/each}}
            </div>
        </div>
        {{/each}}
    </div>
</script>

When I combine all that with my data, I get something like:

<div id="league-accordion" class="panel-group">
    <div class="league-header">
        <h1>National Hockey League (NHL)</h1>
    </div>
    <h3>Founded 1917</h3>
    <div class="panel panel-default">
    <div class="teamDiv">
        <h3>
            Colorado Avalanche
        </h3>
    </div>
    <div class="teamDiv">
        <h3>
            Chicago Blackhawks
        </h3>
    </div>
    <div class="teamDiv">
        <h3>
            Columbus Blue Jackets
        </h3>
    </div>
    <div class="teamDiv">
        <h3>
            St. Louis Blues
        </h3>
    </div>
    <div class="teamDiv">
        <h3>
            Boston Bruins
        </h3>
    </div>
    <div class="teamDiv">
        <h3>
            Montreal Canadians
        </h3>
    </div>
    <div class="teamDiv">
        <h3>
            Vancouver Canucks
        </h3>
    </div>
    <div class="teamDiv">
        <h3>
            Washington Capitals
        </h3>
    </div>
    <div class="teamDiv">
        <h3>
            Phoenix Coyotes
        </h3>
    </div>
    <div class="teamDiv">
        <h3>
            New Jersey Devils
        </h3>
    </div>
          ...      
    </div>
</div>

The power of this becomes more obvious as I try to sort my data with the user helpers…..in my next blog.

Basic Javascript Templates with Handlebars

I have recently worked on a project using Javascript with Handlebars as a templating engine. I do not claim to be a Handlebar, or Javascript, expert, I am just making note of what I have learned for future reference and maybe something I write here will benefit someone else who trying to get started with Handlebars.

I have made a simple little example project with some sports themed data. My data is a collection of sports leagues. Each league has a Name, an acronym, the year it was founded, and can have a collection of teams. Each team has a city, a name, and a division. The structure in Json form is this:

var sportData = {
    league: [
        { name: "National Hockey League", acronym: "NHL", founded: 1917, },
	{ name: "Major League Baseball", acronym: "MLB", founded: 1869, },
        { name: "National Basketball Association", acronym: "NBA", founded: 1946, },
        { name: "National Football League", acronym: "NFL", founded: 1922, },
        { name: "Major League Soccer", acronym: "MLS", founded: 1993, },
    ]
};

In theory this is something that is returned from an API, but for the purpose of this small example, in code will work. So the question is, how can I easily display this data? The answer that I am covering in this blog is……Handlebars. Shocker, right?

So here is my simple Handlebars template to plop that data into the web page for me.

<script id="sports-template" type="text/x-handlebars-template">
    <div id="sport-list">
        <ul>
            {{#each league}}
            <li>
                <label>{{{name}}} ({{acronym}}) [Founded {{{founded}}}]</label>

            </li>
            {{/each}}
        </ul>
    </div>
</script>

So let’s break this down. This really is just HTML with some handlebars bits injected. So I have a container div I call sport-list. This literally is just a wrapper div that I choose to put. It is not used for anything at the moment. Next I have a <ul> to start my list. Just below that, there is the first bit of handlebars notation, which is {{#each league}}. This is essentially a foreach that is seen in many other languages. So {{#each league}} will look for an object called league within the current data context. Then, for each object within the league object, handlebars will repeat the chunk of code until the matching {{/each}}. Inside of the each, there is an <li> just to make each entry its own list item, then a boat load of curly braces. The first handlebars bit is {{{name}}}, which tells handlebars to put whatever is in the object called “name” there. Then {{{acronym}}} tells handlebars to put whatever is in the object call “acronym” in that place. I am sure that you can now guess what {{{founded}}} does. If any of that data is missing, handlebars essentially leaves it blank.  The last bit of important information is the id on the script tag. That will be what is given to handlebars to let it know what template to use.

Before you can use a handlebars template on a webpage, it needs to compiled. To do that, we have to call “Handlebars.compile()” and pass in the id of the script that you want compiled, in my case that is “sports-template.” I have chosen to put the actual Handlebars.compile call inside of an object called “handlebarsSample” that will also be used to hold as much of the handlebars related javascript as possible. “handlebarsSample” looks like this:

var handlebarsSample = function () {
    function compileTemplate(scriptId) {
        if (scriptId[0] != "#") {
            scriptId = "#" + scriptId;
        }

        return Handlebars.compile($(scriptId).html());
    }

    function registerPartials() {
        //No partials right now
    }

    function registerHelpers() {
        //No Helpers right now
    }

    return {
        compileTemplate: compileTemplate,
        registerPartials: registerPartials,
        registerHelpers: registerHelpers,
    };
}();

To explain this, I will start at the top. The first thing I do is define an object called “handlebarSample,” which is just a function used to house other objects. Currently, those three objects are registerPartials, registerHelpers, and compileTemplate. The two registers are not currently being used because my sample is too simple to need them yet, so I am not going to explain them. They are just placeholders at the moment. compileTemplate, however, is used. That is where I am actually calling the Handlerbars.compile function. This allows me to do some validation or error checking of the script Id that I want to compile. My case I am just making sure that I have the needed “#” that jquery uses to find Ids in the DOM. If the first character is not a “#” put one, then pass that to the compile function, which is returned from this function to where ever the call to it was made.

So now I have data. I have made a template. I have compiled that template. But I have nothing showing up on my webpage yet.  That is because we have not yet plugged the data and template together, nor have we injected the result. The first step is to actually compile the template.

var sampleTemplate = handlebarsSample.compileTemplate("sports-template");

Then I want to take that result, plug the data into, and put it somewhere on the page.

$('#sample-div').append(sampleTemplate(sportData));

Lets work inside-out for this. The sampleTemplate(sportData) is where I plug my data into the handlebars template. Then I use a basic jquery selector to grab the appropriate div, then I just append the results that div which gives the output of:

National Hockey League (NHL) [Founded 1917]
Major League Baseball (MLB) [Founded 1869]
National Basketball Association (NBA) [Founded 1946]
National Football League (NFL) [Founded 1922]
Major League Soccer (MLS) [Founded 1993]

That is the very basics using handlebars. There is a more I plan to cover, including partials and helpers in future blogs, hopefully soon (no promises).

For the full source look at these two files.
SportsLeagues.html
handlebarsSample.js

People Watching, with SoftKinetic and Iisu

I recently spent some time working on project that utilizes SoftKinetic cameras through the Iisu SDK, more specifically the Iisu.net wrapper. For those of you who don’t know what SoftKinetic is, the cameras operate similar to Kinect. The cameras attempt to detect and track people who are in their field view. This allows me, the developer, to create applications that react to the user without the need of a tradition input devices (mouse, keyboard, etc.) This it typically done though known gestures, but does not have to be.

Getting Started

The first step in using the Iisu SDK, is getting a device that is Iisu compatible. For my most recent project, I was working with the Panasonic EKL3106 and the SoftKinetic DS311. The EKL3106, the DS311 and the Kinect are very similar in appearance, and actually very similar in function and what data they each can provide. When using the EKL3106 and DS311 in conjunction with the Iisu SDK, I was able to detect and track people, as well as recognize gestures. The Iisu SDK is available through the SoftKinetic website, and does require licensing.

Development Experience

Getting started with Iisu.net was time consuming, and a bit frustrating. Currently for Iisu.net, the documentation is very limited. The first lesson I had to learn was to not rely on the C# documentation; instead look at the C++ documentation, and learn how to translate that information to work for your purposes in the C#/.Net world. The C# documentation’s usefulness really was limited to getting a basic connection to the hardware, beyond that, the C++ documentation was all I could rely on. The process of starting Iisu.net and the camera relies on the use of case sensitive string literals. This makes the learning process of what can be done much more difficult. Enabling a feature of the camera, or subscribing to some data feed from the camera, required me to look in the documentation for the exact string to pass in. Although not difficult, just not as intuitive as I had hoped and required a bit of investigation. Once that was sorted out, the Iisu.net wrapper provides an easy interface to get data from the camera.

Usability

I feel the user as a controller technology still has a lot of room to improve. Too often the camera would report false positives and would track a person that was not there. This prevented the camera from detecting when there was an actual user in front of the camera. The opposite problem also occurs. The camera sometimes has difficultly detecting when a person is in front of it. In general, tracking people needs to get more reliable. The Iisu needs to be able to see at least from head to waist to detect a user, and is most accurate when place above head level looking down. That being said, once a user is detected, the tracking of various joints is pretty accurate, and often to an impressive degree. Another positive of the Iisu SDK, is the built in gesture recognition. The Iisu SDK has a concept of a controller built in. What this means, is that if a person is in front of the camera and performs a known gesture with their hand (wave or circle) a “controller” at the gesture location. This feature allows me to track the users hand, as a controller, pretty accurately. This method is the route I went in the my most recent project and works very well. It makes my life much easier to do simple hand tracking and gesture recognition.

I like the idea of touchless interaction and the “magical” feel it provides, but after using the technology, it needs to mature a bit in order to get more reliable. I am excited to see what the next generation of hardware will bring.