Serving Python ML models behind a Node API

Getting a model into production

A trained model is not a product until something can call it. There are two patterns worth knowing, and the wrong one will cost you uptime.

Pattern A: a separate inference service

Keep Python and Node in separate processes. Serve the model from FastAPI and have your Node API call it over HTTP.

app.post('/api/predict', async (req, res) => {
  const response = await fetch(process.env.INFERENCE_URL + '/predict', {
    method: 'POST',
    body: JSON.stringify(req.body),
    headers: { 'Content-Type': 'application/json' },
  });
  const prediction = await response.json();
  res.status(200).json({ success: true, data: prediction });
});

This is the default choice. The two services scale independently, a model reload never restarts your API, and a Python crash does not take Node with it.

Pattern B: spawning a subprocess

app.post('/api/classify', (req, res) => { const proc = spawn('python', ['scripts/classifier.py', req.body.text]); let out = ''; proc.stdout.on('data', (d) => { out += d; }); proc.on('close', () => res.json({ success: true, classification: out })); }); ```

Only reach for this on a single box with low traffic. Every request pays full interpreter startup and reloads the model weights, which is usually far slower than the inference itself.

Which to pick

If the model is larger than a few hundred megabytes, or you expect more than a handful of requests per second, use Pattern A. We have never regretted the extra service; we have regretted the subprocess.