Azure Kinects calibration

Kinect left  ─► Kinect Azure CHOP ─► transformxyz_Y2  (pose left)  ─┐
Kinect right ─► Kinect Azure CHOP ─► transformxyz_Y19 (pose right) ─┼─► fusion (Script CHOP) ─► OSC Out ─► lab network
Kinect rear  ─► Kinect Azure CHOP ─► transformxyz_Y13 (pose rear)  ─┘

There is no selection, renaming or smoothing between these nodes. All of the decision making happens in one place: the Script CHOP fusion inside Sensor_Azure_Multiple_HOST, with its code in the DAT fusion_callbacks. Smoothing is done on the client, not here, so the host adds no delay.

Each Kinect Azure CHOP runs Microsoft's body tracking for its own sensor. Relevant settings:

• World Space Positions on. Joint positions in metres, relative to that sensor.
• Bone Confidence on. Adds a :confidence channel per joint (see Fusion per joint).
• Max Players. The number of bodies one sensor tracks. Each extra body costs GPU time on the NUC.

Per tracked body, a sensor outputs p{k}/id, and for each of its 32 joints p{k}/{joint}:tx/ty/tz/confidence. The k here is the sensor's own body number. It means nothing outside that sensor (see Decision making).

The sensors hang on drop rods below the pipe grid, 3.03 m above the floor. All three face the centre wall and tilt down by about 45°.

Each pose is stored in a Transform XYZ CHOP per sensor (transform order Scale Rotate Translate, rotate order Rx Ry Rz). The Transform CHOP turns that sensor's skeleton data from sensor coordinates into room coordinates. Calibration therefore comes down to these six values per sensor. If a sensor is moved, only its own Transform CHOP needs updating. The poses are also shown in the Lab model.

Calibration is what makes the decision making possible: only after this step can a position measured by one sensor be compared with a position measured by another. Any remaining calibration error shows up as the same person being placed in slightly different spots by different sensors. In a test walk (25 September 2026) that difference was 0.1–0.5 m, with the largest differences always involving the left sensor (about 0.3 m in X and 0.1 m in Y).

Microsoft's body tracking runs separately on each sensor, and each sensor numbers the bodies it sees on its own. "Body 1" on the left sensor and "body 1" on the right sensor are unrelated labels. The same person usually has a different number on each sensor, and that number changes when they leave a sensor's view and come back.

An earlier version of the host worked around this by picking one sensor's data per moment ("switch phases" based on margins). That broke as soon as more than one person was in the room, or someone walked from one sensor's field of view into another's.

The host therefore ignores the sensors' own body numbers and decides identities itself, based only on where bodies are in the room. Every frame, fusion answers four questions:

1. Which detections are real?
2. Which detections belong to a person who is already known?
3. Which of the remaining detections are new people, and are they real?
4. For each joint of each person, which sensors should be trusted?

The answers are kept between frames as a list of tracks. A track is one person: their number (or none yet), their last fused pose, how many frames in a row they were seen, and how many frames they have been missed.

From each of the three inputs, every body slot is read. A body counts as a detection only if:

• its id is greater than 0 (the Kinect reports 0 for an empty slot), and
• it has a pelvis joint.

Positions can't be used to recognise an empty slot, because the calibration transform turns the sensor's zeros into the sensor's own position in the room.

The pelvis is used as the reference point for all comparisons in steps 2 and 3. It is the root of the skeleton, the most stable joint, and the one the Kinect rates with the highest confidence.

Every detection is first compared with every existing track, using the distance between the detection's pelvis and the track's pelvis from the previous frame. All pairs are sorted from closest to farthest and handed out in that order (a greedy nearest-neighbour assignment):

• A pair is accepted if the distance is under MATCH_DIST = 0.75 m. That is far more than anyone moves in one frame, and less than the usual distance between two people.
• One body per sensor per track. A sensor can't contribute two bodies to the same person.
• One track per body. Once a detection is used, it's gone.
• On equal distance, a confirmed person wins over a candidate that has no number yet.

Then a second pass catches sensors that disagree more because of calibration error: a detection that is still unused may join a confirmed person within MERGE_DIST = 1.20 m, as long as that person has no body from that sensor yet. Without this pass, a badly calibrated sensor would keep creating a second, ghost person next to the real one.

Why track-first. The first version did it the other way around: it grouped the sensors' bodies into people first, and compared those groups with the previous frame afterwards. That caused numbers to jump (p1 → p2 → p3) every time a sensor picked someone up or lost them. When a sensor joined, the combined position shifted, or a sensor that briefly disagreed formed its own group and "stole" the number. In the current order, a sensor joining or leaving only adds a body to an existing person or takes one away. It can never create a new number for someone who is already tracked.

Only detections that matched no track can start a new person. They are grouped first: a detection joins the nearest group whose average pelvis is within MERGE_DIST, provided that group has no body from that sensor yet. Each group becomes a candidate: a track without a number.

A candidate gets a number only after it has been seen in CONFIRM_FRAMES = 5 frames. A candidate missed for more than TENTATIVE_DROP = 2 frames is discarded. This filters out one-frame glitches, such as a coat on a chair briefly recognised as a body, so they never take a number or reach the clients.

Which number. A confirmed candidate gets the lowest free number from 1 to MAX_SLOTS = 8. The first person in an empty room is always p1. That way a client set to Person 1 always gets someone as long as anyone is in the room. When a person leaves, the others keep their numbers: nobody is renumbered, and the free number goes to the next person who enters.

A confirmed person who isn't matched in a frame keeps their last fused pose and their number. They are released only after DROP_FRAMES = 15 frames in a row without a match (about 0.5 s at the host's 30 fps). A short occlusion, such as walking behind another person, therefore doesn't cost anyone their number.

For each person, every joint is fused from the bodies attached to that person in this frame. The Kinect rates every joint: 0 = none (out of range), 1 = low (occluded, predicted), 2 = medium. The decision per joint:

1. Find the best confidence level any of the sensors reports for this joint.
2. Use only the sensors at that level, and ignore the rest.
3. Take their weighted average (weights: level 2 = 1.0, level 1 = 0.3).

If every sensor rates a joint 0, it is still averaged across all sensors rather than frozen. This matters for hands: the Kinect very often rates hand, hand tip and thumb as 0, even when the hand is clearly visible. An earlier version kept the last known position for such joints, and hands froze in place (in one test 2.5 m behind the body).

Only confirmed people are sent, in number order:

The presence flags and the heartbeat exist because of how OSC behaves on the receiving side: a channel that stops arriving keeps showing its last value forever. Without them, a client can't tell a person who left from one standing still, or a crashed host from a quiet room.

Two people, A and B, in an empty room:

1. A walks in on the left. The left sensor sees one body. No tracks exist, so it becomes a candidate. After 5 frames it becomes p1.
2. A walks to the middle. The rear sensor now also sees A, 0.3 m from where the left sensor places A. In step 2 that body is within 0.75 m of p1, so it simply joins p1. A's pelvis is now fused from two sensors.
3. B enters on the right, 3 m from A. The right sensor's body is far from p1 (no match), so it becomes a candidate and, 5 frames later, p2.
4. A leaves. After 15 frames without a match, p1 is released and p1/present goes to 0. B stays p2.
5. C enters. C gets the lowest free number: p1.

All constants are at the top of fusion_callbacks:

With DEBUG = True, the Textport prints once per second how many bodies each sensor sees and every track as (number, frames missed), where 0 means a candidate without a number. For every person seen by more than one sensor, it also prints where each sensor puts their pelvis and the spread between them: a direct measurement of the calibration. Under about 0.3 m is good. Regularly above 0.5 m means a Transform needs re-measuring; the line shows which sensor differs.

• Identities come from positions in the room, not from any single sensor. That makes the method independent of the hardware: the same code works with Azure Kinects or with a successor such as an Orbbec sensor (see Decision making: sensor succession).
• Three overlapping views mean fewer dropouts when one sensor's view is blocked, and confidence-based fusion takes each joint from the sensors that see it best.
• Track-first matching keeps numbers stable while people walk between sensor views.
• The host never filters for a user. Everything is sent; clients choose. One stream serves any number of projects.

All traffic stays on the lab's local network. Commands from the workstation are sent as broadcasts, so the components don't need to know the host's IP address.

What the client does with this stream (choosing a person and joints, detecting stale data, smoothing, sharing one receiver between copies) is described in Azure Kinect sensors.