Tom Ngo's Embedded Constraint Graphics (Interval Research, 1996, patent US5933150, expired 2016). You draw a few examples of a picture. Then you drag the picture itself, and the engine works out which blend of your examples puts the part you're holding under your cursor.
It's direct manipulation for anything you can show by example. You don't hunt for the slider that does what you want; you grab what you want to move and move it. An example is any list of numbers, so the same engine applies to:
- Facial animation: drag a mouth corner or an eyelid and the face finds the expression. The expressions can be ARKit blendshape coefficients from a rig built in Blender.
- Character animation: skeleton poses instead of faces. Record a performance as weights over time and play it on another character.
- 2D and 3D graphics: cartoon outlines, colors, a 3D head you turn by the nose.
- User interfaces: example layouts of windows, so dragging one window rearranges the rest.
Language models mix ideas the same way, with weights that sum to one.
This page starts with a face that talks, takes it apart one idea at a time, then points the same engine at each of these. The end covers where it could live, and how to take it: the idea is what's free, and the code is one worked example of it.
Every demo is live, and beside each drawing is the machinery Tom's editor kept hidden: the weights, the map of the state space, the shape of the matrix the solver inverts. Drag the drawing, the dot on the map, or a weight bar. Click anywhere on a map to travel there.
Mouther
Golan Levin's Mouther (Interval Research, 1995, with Malcolm Slaney and Tom Ngo) listened to your voice and drove a cartoon face through the sounds you made, in real time. The face was built with Tom Ngo's Embedded Constraint Graphics: ten drawings of a mouth, and an engine that blends between them.
This face has ten mouths, M, OO, R, OH, UH, L, AY, EE, F and S, each a few numbers (how wide, how open, how round, how much teeth and tongue) turned into twenty points.
Four ways to move it, all moving the same weights:
- Pull a face. Smile, grin, laugh, smirk, sad, cry, angry, disgust, surprise, kiss, raspberry, yawn. Each is one more drawing of the same twenty points. From any mouth it eases back through M and out to the face.
- Say. Type something and press Say. It speaks, and the mouth follows the letters, re-syncing on every word.
- Listen. Press Listen and talk. Loudness opens the mouth; how bright the sound is picks the shape, dark OO and R to bright EE and S. Your voice drags the dot on the map.
- Pull. Grab the lips, teeth or brows and drag. The engine finds the blend of mouths that puts the point under your finger, and switches between them as you go. Or drag the dot on the map, or one weight bar.
The map is the face's state space. M, lips pressed, is in the middle; the nine sounds sit around it, dark to bright, and every triangle of M and two neighbours is a zone. A point in a triangle is a blend of three mouths. The twelve faces make a second fan around the same M. Both fans can't lie flat in one picture, so the faces get their own page, glued to the sounds at M: when the state is at rest, the dot shows on both pages, and you can grab it on either. Driving (Say, Listen) and tugging (Pull) are two ways of moving that one point.
How that works, one idea at a time, is the rest of this story, starting from Two Pictures.
The mouths come from examples/ecg/tools/mouther.py; change a number and run it to restyle the face.
How it works
Everything Mouther does, built up from nothing. Each step adds one idea.
Two Pictures
Two drawings of a mouth with the same three points: a frown and a smile. Any mix of the two is a mouth too. The mix is two weights that add up to 1, and the drawing is each point at its weighted average.
Click frown and smile in the panel to drive the blend. Then grab a corner of the mouth and pull it up or down. The engine finds the weights that bring the corner closest to your cursor; the other two points come along.
Two examples make a line segment of blends. Pull past either end and the mouth stops there: there's no blend beyond 100% smile.
The state in a line segment.
| 1.000 | ||
| 0.000 |
Drag a point, or the dot on the map. Shift-click pins a point. The buttons drive to an example.
The readout says J is 2×1, tall: you asked for two numbers (the corner's x and y) and there's one free number (how much smile). You can't always get both, so the solver gets as close as it can.
A Triangle of Pictures
One point, at a different place in each of three examples A, B and C. Three examples make a triangle of blends, and the point's three positions happen to draw that triangle on screen.
Drag the point. Inside the triangle it follows the cursor exactly: J is 2×2, square, one exact answer, and the weights are the cursor's barycentric coordinates.
The state in a triangle.
| 1.000 | ||
| 0.000 | ||
| 0.000 |
Drag a point, or the dot on the map. Shift-click pins a point. The buttons drive to an example.
Try: drag to (2, 3) and read the weights: 0.5, 0.2, 0.3.
Now drag out past an edge. A weight would have to go below zero ("less than none of B"), so the engine pins it at zero and the state slides along the wall. The crosshair is where you asked for; the point is the nearest blend. Keep going past a corner and everything stops at that one example.
More Examples Than You Need
Add a fourth example, D. Four examples make a tetrahedron of blends, but the point still lives on a flat screen, so many different blends put it in the same place.
Drag it. J is 2×3, wide: two numbers asked for, three free. The solver picks the answer that changes the weights least, so the state stays near where it was. Watch the weights: dragging the same path from A and from D gives the same point and different blends.
The state in a tetrahedron, seen down three axes.
| 1.000 | ||
| 0.000 | ||
| 0.000 | ||
| 0.000 |
Drag a point, or the dot on the map. Shift-click pins a point. The buttons drive to an example.
The free choices left over are the null space: weight changes that don't move the point. A later step spends them on something useful.
More Handles Than You Can Satisfy
A face: three points of mouth, two of each brow, four expressions blended in one tetrahedron.
Grab a mouth corner and pull it up: the face smiles, and the brows lift too, because in these examples they move with the mouth. Grab a brow and push it down: the face frowns.
The state in a tetrahedron, seen down three axes.
| 1.000 | ||
| 0.000 | ||
| 0.000 | ||
| 0.000 |
Drag a point, or the dot on the map. Shift-click pins a point. The buttons drive to an example.
Now shift-click both inner brow points to pin them, and pull the mouth again. Each pin adds two rows to J; with three handles J is 6×3, tall, and the solver splits the difference. Pull the mouth into a big smile with the brows pinned low and you get something sly that isn't any of the four examples.
On a touch screen, drag two points at once with two fingers. It's the same stacking.
Gluing Triangles Together
One big simplex blends everything with everything, which is rarely what an artist wants: a frown mixed with a smile is mush. So Tom glues small simplices, zones, along shared faces. Here the face has two zones that share the edge neutral–surprise:
- neutral, smile, surprise
- neutral, surprise, frown
In the panel, the current zone is shaded. Start at smile and drag the mouth's middle up into a frown. The state runs into the boundary between the zones, steps across at exactly the blend it reached (Tom's via point: the shared examples keep their proportions, the new one starts at zero), and keeps going. The face never jumps, and it never passes through a smile-frown blend, because no zone contains both.
In zone neutral–smile–surprise, a triangle, one of 2.
| 0.000 | ||
| 1.000 | ||
| 0.000 | ||
| 0.000 |
Drag a point, or the dot on the map. Shift-click pins a point. The buttons drive to an example.
Under the panel, the readout names each crossing: which zone it left, which it entered, and through which face. Drag the mouth's middle straight up and it crosses at neutral itself: a face can be a single shared example, not only a whole edge.
A Fork in the Complex
A kite and its tail. Examples A and B hold the kite low on either side; C flies it high; D and E are two different ways to dive, with the tail swinging differently. Three zones share the edge AB: ABC above, ABD and ABE below.
Start at C and drag the kite straight down past AB. Both ABD and ABE can put the kite under your cursor, so the engine has to pick one. Tom suggested two rules, and the chooser offers both:
- least state movement: the zone where the weights change least for this drag
- least movement of unheld points: the zone where the points you aren't holding (here, the tail) move least
Drag down with one rule, press Reset, switch the rule, and drag down again. The readout shows each candidate's score.
In zone A–B–C, a triangle, one of 3.
| 0.000 | ||
| 0.000 | ||
| 1.000 | ||
| 0.000 | ||
| 0.000 |
Drag a point, or the dot on the map. Shift-click pins a point. The buttons drive to an example.
Neither rule is always right. That's why it's a knob.
Walking
An inchworm has three poses: lying flat, humped with its tail pulled up behind, and reaching forward with its head. They're joined in a loop of edges: flat to humped, humped to reaching, reaching back to flat.
The trick is on the last edge. Going from reaching to flat, the flat pose is shifted one stride forward. Tom calls this an automatic transformation: each edge can carry one, and walking the edge backwards undoes it.
Drag the hump to the right. The worm humps, reaches, lies flat a stride further on, and does it again as long as you keep dragging. Drag left and it backs up.
In zone flat–humped, a line segment, one of 3.
| 1.000 | ||
| 0.000 | ||
| 0.000 |
Drag a point, or the dot on the map. Shift-click pins a point. The buttons drive to an example.
Only the hump (the middle point) moves steadily forward in every pose, so it's the handle that walks. The head stands still while the tail catches up, and the tail stands still while the head reaches. Drag those and see.
In the panel, the dashed edge is the one carrying the stride. The state goes round the loop once per step.
Tumbling
The same trick, rolling instead of crawling. A box has three poses: sitting flat, tilted 30° up onto its bottom-right corner, and tilted 60°. Past 60° it lands flat again, one box-width along.
But it lands on its side: the corner it pivoted on is now bottom-left, and the next roll has to pivot on the corner that was top-right. So the edge from 60° back to flat carries "one box-width right, and rename the corners". Renaming points is a linear map on the drawing like any other, so an edge can carry it, and walking the edge backwards renames them back.
Push the box to the right by a top corner. The bottom-right corner is the pivot, so it can't push this roll; after the roll, a different corner is the pivot. Push it left and it rolls back.
In zone flat–30°, a line segment, one of 3.
| 1.000 | ||
| 0.000 | ||
| 0.000 |
Drag a point, or the dot on the map. Shift-click pins a point. The buttons drive to an example.
Look closely mid-roll: the box shrinks a little. Each pose is placed once when the state enters a zone, and then the placed poses are blended linearly, so a corner cuts across the chord of its arc instead of following it. That's Tom's linearized configuration map. It costs a little accuracy and keeps the drag a small linear solve. More poses make the chords shorter.
Feel
The patent says what the state is. It doesn't say how dragging it should feel. That part you learn with your hands.
| Knob | What it does | What it costs |
|---|---|---|
| damping | calms jitter when examples nearly agree | the point lags the cursor |
| snap | lands exactly on an example when close | hard to stop just short of one |
| spring | eases back to the nearest example on release | can feel floaty |
| inertia | a flick keeps going after you let go | overshoots what you aimed at |
| overshoot | lets weights dip below zero at a wall, then springs back | too much looks broken |
| sticky | a detent at the face between zones | notchy if overdone |
Try the presets, then make your own. Rubber stretches past the walls and wobbles back. Clay stays where you leave it. Puppet keeps moving when you flick it and clicks between zones.
In zone neutral–smile–surprise, a triangle, one of 2.
| 1.000 | ||
| 0.000 | ||
| 0.000 | ||
| 0.000 |
Drag a point, or the dot on the map. Shift-click pins a point. The buttons drive to an example.
Color
Anything you can add up and scale can ride along on the same weights as the points. Here each example has a color as well as a shape: a red square, a yellow diamond, a blue pillar. Drag a corner and the blob changes shape and color together, because it's one blend.
The state in a triangle.
| 1.000 | ||
| 0.000 | ||
| 0.000 |
Drag a point, or the dot on the map. Shift-click pins a point. The buttons drive to an example.
Drag halfway between red and blue, then switch blend in from OKLab to raw sRGB. Averaging raw sRGB numbers sags darker in the middle: perceived lightness 0.61 at red and 0.64 at blue, but 0.57 halfway, against 0.63 in OKLab. Equal steps in sRGB aren't equal steps to your eye. OKLab (Björn Ottosson, 2020) is built so they are. Same weights, same simplex; only the space the color numbers live in changes.
Motion, Skeleton, Skin
Tom splits an animation into three parts you can swap:
- the clip motion: how the state moves over time, a path through the simplices
- the graphic skeleton: one character's examples, turning a state into points
- the graphic skin: how the points are drawn
A person and a cat, drawn separately, with the same four expressions under the same names. Their points have nothing in common: the cat has ears and no brows. But they share a state space, so one state drives both. Drag either one.
Pull the cat's ears up and the person looks surprised, though the person has no ears. The cat's ears and the person's brows are different ways of saying the same thing, and the state is the thing they say.
So one state can drive any number of characters, each with its own way of showing it. A dog drops its ears where a cat flattens them; a bird puffs its feathers. Add an expression called lie, and Pinocchio's nose grows while Nixon's jowls give the tell.
Then press Record, perform something on the person, press Stop and Play. The motion is only weights over time, so it plays on the cat too.
In zone neutral–smile–surprise, a triangle, one of 2.
| 1.000 | ||
| 0.000 | ||
| 0.000 | ||
| 0.000 |
Drag a point, or the dot on the map. Shift-click pins a point. The buttons drive to an example.
The examples don't have to be drawings. They can be skeleton poses: a body's joint angles at the corners, and the state a blend of postures. Ken Perlin's Improv (with Athomas Goldberg, NYU Media Research Lab, SIGGRAPH 96) drove interactive actors that way, layering weighted actions on a skeleton and adding noise so they never stood dead still. Don Hopkins built the character animation system of The Sims 1 inspired by Improv.
This is how Mouther works: the voice drives the state, and the face is a skin.
The same engine, other things
The engine never knows what the numbers mean. It needs two things: examples that are lists of numbers of the same length, and handles whose positions are linear in those numbers. A point of a drawing is two of the numbers. A point seen through a fixed camera is a linear function of three. A rig's mesh is a linear function of its coefficients. Anything like that, you can drag.
Turning a Head
Nothing in ECG is 2D. An example is a list of numbers, and here each one is a head: points on a sphere, turned in three dimensions, then seen through a camera. Five poses: ahead, turned left, turned right, looking up, looking down.
The camera is fixed and looks straight on, so a point's place on the screen is a linear function of where it is in 3D. Blending 3D poses and then projecting gives the same picture as projecting and then blending, which means the engine can solve on the screen and the head moves in 3D.
Grab the tip of the nose and steer. Up and to the right is a blend of three poses, ahead, up and right, and the map shows you which triangle you're in.
In zone ahead–left–up, a triangle, one of 4.
| 1.000 | ||
| 0.000 | ||
| 0.000 | ||
| 0.000 | ||
| 0.000 |
Drag a point, or the dot on the map. Shift-click pins a point. The buttons drive to an example.
Turn it all the way and watch the eyes: they get a little closer to the middle than a real turn would put them. That's the linearized configuration map from Tumbling again: blended poses cut across the chord of a rotation instead of following the arc. Add poses at 25° and the chords get shorter. A real character uses the same trick with poses of a skeleton, and a renderer draws the mesh the skeleton carries.
The head comes from examples/ecg/tools/worlds.py.
Blendshapes
Film and game faces are rigged with blendshapes: one sculpted offset of the mesh per muscle-ish movement, mixed by coefficients from 0 to 1. Apple's ARKit face tracking reports 52 of them every frame, with names like jawOpen, mouthSmileLeft and eyeBlinkRight, and tools such as FaceIt for Blender build those 52 on any character's mesh. Blender calls them shape keys, glTF calls them morph targets, three.js calls them morph target influences.
This face is rigged with fifteen of them. Each expression is not a drawing but a setting of the coefficients:
| Face | Coefficients |
|---|---|
| smile | mouthSmileLeft 1, mouthSmileRight 1, browOuterUpLeft 0.2, browOuterUpRight 0.2 |
| wink | eyeBlinkLeft 1, mouthSmileLeft 0.8, mouthSmileRight 0.3, browDownLeft 0.3 |
| skeptical | browOuterUpRight 1, browDownLeft 0.7, mouthSmileRight 0.4, mouthFrownLeft 0.3 |
| surprise | jawOpen 0.7, browInnerUp 1, browOuterUpLeft 1, browOuterUpRight 1, eyeWideLeft 1, eyeWideRight 1 |
| kiss | mouthPucker 1, eyeBlinkLeft 0.6, eyeBlinkRight 0.6, browInnerUp 0.3 |
| sad | mouthFrownLeft 1, mouthFrownRight 1, browInnerUp 1, eyeBlinkLeft 0.25, eyeBlinkRight 0.25 |
| angry | browDownLeft 1, browDownRight 1, mouthFrownLeft 0.6, mouthFrownRight 0.6, eyeWideLeft 0.4, eyeWideRight 0.4 |
The rig is linear: the face is the neutral face plus each offset times its coefficient. So blending the expressions' coefficients and then rigging gives exactly the face you get by blending their points. ECG can work on the points you see, and hand the rig the coefficients.
Drag a mouth corner, an eyelid or a brow. The engine finds the blend of expressions that puts it under your finger, and the bars under the weights show the fifteen coefficients that blend works out to: the numbers a game engine or Blender would get.
In zone neutral–smile–wink, a triangle, one of 6.
Drag a point, or the dot on the map. Shift-click pins a point. The buttons drive to an example.
That's the inverse of what rigs usually offer. An animator sets coefficients and watches the face; here you pull the face and get coefficients, and the expressions keep the result looking like a face instead of fifteen sliders' worth of possible faces. J.P. Lewis and Ken Anjyo brought this to production blendshapes in Direct Manipulation Blendshapes (2010); ECG's complex of expressions adds that the faces you blend are chosen, a few at a time, and you walk between them.
The rig comes from examples/ecg/tools/worlds.py.
Bodies
A body is a skeleton, and a pose is its joint angles. Angles don't blend as freely as points (halfway between 350° and 10° is not 180°), so a body blends poses that are near each other, or blends rotations properly, and then hands the result to the skin. Ken Perlin's Improv drove actors this way, and Improv inspired the character animation system Don Hopkins built for The Sims 1.
ECG adds the drag. Author a few postures, glue them into a complex, and pull a hand: the engine finds the blend of postures that puts the hand there, and the rest of the body comes along the way it did in the postures you drew. Because many blends put the hand in the same place, slow noise among them makes the body breathe while the hand stays put.
That's how the next character will move: Heinz Lemke at the PDP-7, holding the light pen against the Type 340 display while his program PIXIE tracks it. The pen tip is the handle, a recorded demo is just a path for the tip, and ECG fills in the arm, the lean and the face (the portrait design). His head, hands and body will be built in Blender, the face rigged with FaceIt's ARKit blendshapes, so one engine drives all of it: the expressions from Mouther-style voice and gesture, the hand from the pen, the body from both.
Arranging Views
A layout of windows is a list of numbers too: each window's corners. Save a few layouts you like and they're examples, and the arrangement on the screen is a blend of them.
Four windows: the yellow article, the green source, the pink memory and the blue map. Four layouts: reading (the article big, source and map beside it), comparing (article and source side by side), debugging (source, memory and map) and overview (all four tiled). A layout that hides a window shrinks it to a point at the edge of the screen, so blending toward that layout shrinks it away.
Grab a corner of a window and drag it. The engine finds the blend of layouts that makes that window that size, and every other window follows the layouts it's heading for. Four layouts glued into one tetrahedron: the map shows it from three sides.
The state in a tetrahedron, seen down three axes.
| 1.000 | ||
| 0.000 | ||
| 0.000 | ||
| 0.000 |
Drag a point, or the dot on the map. Shift-click pins a point. The buttons drive to an example.
Nothing here is about windows. The same numbers could be a camera's position and zoom over a map, the panels open in an emulator, or the scroll position of each article: the views you saved are the examples, and pinching, scrolling or resizing drags you between them. The design for doing this to the whole HyperTIES reader, with "views you were heading toward" to finish a move with one click, is VIEW-BLENDING.md.
The layouts come from examples/ecg/tools/worlds.py.
How language models blend ideas
A large language model does a version of this blend thousands of times for every word it writes.
Attention. An attention head scores every earlier token, turns the scores into weights with softmax, so every weight is positive and they sum to 1, and outputs the weighted sum of those tokens' value vectors. Those weights are barycentric coordinates. The output is a point in the simplex whose corners are the values, exactly where the dot on a map sits among the examples.
Mixture of experts. A router scores every expert, keeps the top two or so, and blends only those. Out of many corners it picks a small simplex to blend in, the way ECG blends three mouths out of twenty-two by choosing a zone.
Ideas as directions. Word embeddings did arithmetic early on: in word2vec (Mikolov and colleagues, 2013), king − man + woman lands near queen. Interpretability work since breaks a model's activations into a sparse sum of learned features, a few active at a time (Anthropic's dictionary learning, 2023–2024), and pushing on one feature steers what the model says: clamp the Golden Gate Bridge feature up and the model brings the bridge into everything (Golden Gate Claude, 2024). The same concept in different languages lights up shared features (On the Biology of a Large Language Model, 2025).
That's the person and the cat from Motion, Skeleton, Skin. Surprise is one state; the person shows it with brows and the cat with ears. A concept is one direction; English and French each give it their own words.
ECG's move is the inverse problem: here is the output I want, which blend of things you know produces it? Steering and feature attribution ask the same question of a model. The difference is size. ECG's corners are a dozen pictures a person chose, and its space fits on a map you can drag. That makes it a good place to get a feel for what the big ones are doing.
Where it could live
The forward half of ECG, blending examples by weights, is already built into the tools people use:
- Godot's
AnimationNodeBlendSpace2Dtriangulates its animation samples and blends inside a triangle by barycentric weights. Its zones are ECG's zones. - Blender mixes shape keys by value and drives them from bones and expressions; FaceIt builds ARKit's 52 on any face.
- Illustrator's Blend tool makes in-between steps from one shape to another.
- D3 interpolates between states of a chart; D2 lays out the same diagram several ways.
- Games blend animation clips, cameras and character-creator morphs.
- User interfaces blend layouts, zoom levels and menus.
What's missing in all of them is the other half: grab the result and say where it should go. A Godot blend space you steer by dragging the character's hand. A Blender add-on where you pull a vertex on a FaceIt head and the expression shape keys follow. An Illustrator blend of five drawings you reshape by dragging the in-between. A D3 or D2 diagram whose saved layouts you move between by dragging one node. Window layouts that follow a resize, as in Arranging Views.
Take it
What's free is a way of looking at things: a few examples make a space you can move through, and the thing you see is the control. You don't need this code to use it.
The code is one worked example, and worked examples are what other implementations interpolate from, whether a person writes them or a language model does. Give a model this page and the package and it treats them the way ECG treats examples: as points to blend toward a version in GDScript, Blender Python, C#, or whatever you use.
The engine is packages/ecg: TypeScript, MIT licensed, no runtime dependencies, with a worked test for every idea on this page. Copy it into your project. Tom's patent expired in 2016, so the technique is free to use too.
import { loadModel, startState, walk, pointTargets, configurationOf } from '@wwsff/ecg';
const { model } = loadModel({
examples: [
{ id: 'frown', points: [[-10, 0], [0, -4], [10, 0]] },
{ id: 'smile', points: [[-10, 0], [0, 4], [10, 0]] }
]
});
let state = startState(model);
// the user drags point 1, the middle of the mouth, to (0, 2)
state = walk(model, state, pointTargets(1, 0, 2)).state;
state.weights; // [0.25, 0.75]
configurationOf(model, state); // [-10, 0, 0, 2, 10, 0]
Every demo on this page is an ecg block of plain YAML: examples, zones, strokes. Copy one, change the numbers, and you have your own. The generators for Mouther and the head, blendshape and window demos are short Python scripts beside the articles.
Build it into the tool you use. Rig a face, steer a blend space, arrange a dashboard, bend an icon. Write about what broke. We'd like to see it.
Where it comes from
Tom Ngo and Bruce Donald patented ECG at Interval Research (filed 1996). The SIGGRAPH 2000 paper, Accessible Animation and Customizable Graphics via Simplicial Configuration Modeling, is by Ngo, Cutrell, Dana, Donald, Loeb and Zhu. Golan Levin built Mouther with it, with Malcolm Slaney and Tom Ngo. Blending authored poses to drive a character runs through Ken Perlin's Improv (with Athomas Goldberg, NYU Media Research Lab, SIGGRAPH 96). J.P. Lewis and Ken Anjyo brought direct manipulation to production blendshapes (Direct Manipulation Blendshapes, 2010). More papers: REFERENCES.md.
The history: Tom Ngo's Embedded Constraint Graphics at Interval.