Learn With Champak

Live Facial Expression Recognition with TensorFlow.js

Use a webcam, BlazeFace and FaceMesh to detect faces, locate facial landmarks and study visible facial geometry directly inside the browser.

What does this application do?

The application opens the webcam and runs a two-stage computer-vision process. BlazeFace first detects the faces in the video. When a face is found, FaceMesh estimates detailed facial landmarks. The program measures visible geometry around the eyes, eyebrows and mouth to display simple labels.

Webcam → BlazeFace detection → FaceMesh landmarks → Geometry measurements → Expression-like label

Run the complete application

Select Load TensorFlow Models. After both models are ready, select Start Camera and permit camera access. You can stop the camera or save the displayed frame as a PNG image.

Privacy note: Camera processing runs inside the browser. Grant camera permission only when you are comfortable doing so, and stop the camera after completing the activity.

Open the Application in a New Tab

The two-stage inference process

Stage 1: Detect faces with BlazeFace

BlazeFace searches the webcam frame and returns a bounding area for every face it detects. The application draws a box around each detected face.

Stage 2: Estimate landmarks with FaceMesh

When at least one face exists, FaceMesh estimates facial landmarks. These points provide coordinates for features such as the eyes, eyebrows, mouth corners and lips.

Running FaceMesh only after BlazeFace finds a face avoids unnecessary landmark processing when nobody is visible.

How the labels are calculated

The program calculates distances between selected FaceMesh landmarks. Measurements are divided by the distance between the eyes so that the ratios remain more stable when the person moves nearer to or farther from the camera.

Smile-like
Influenced by mouth width and the upward position of the mouth corners.
Surprise-like
Influenced by mouth opening and eyebrow distance.
Neutral-like
Used when the other geometry scores are comparatively low.

Important limitation

The displayed labels are based only on visible facial geometry. They are not facts about a person’s emotions, thoughts, intentions or mental state. A facial movement can have many causes. This application is an educational demonstration of face detection, facial landmarks, geometry and browser-based inference.

Program features

  • TensorFlow.js with the WebGL backend
  • Separate model-loading and camera-start controls
  • Live face boxes, landmarks, labels and scores
  • Support for as many as five faces in FaceMesh
  • Mirrored webcam display and matching canvas coordinates
  • Start, stop and error-handling logic
  • PNG saving with the webcam frame and overlay
  • Live reporting of detected and landmarked faces

Practice and investigation

  1. Identify the landmark indices used for the eyes, eyebrows and mouth.
  2. Explain why measurements are divided by the distance between the eyes.
  3. Change the geometry thresholds and observe how the labels respond.
  4. Compare performance with one face and several faces.
  5. Use the Save Image button and verify that the overlays appear in the PNG.
  6. Explain why these labels should not be treated as actual emotion recognition.

Learn With Champak

TensorFlow.js, JavaScript, Machine Learning and Computer Vision
Programmer’s Picnic | Champak Roy