
When building APIs we often want to hide or obfuscate certain information from the end user. The most common of which is probably hiding IDs.

## Why hide IDs?

There are many reasons you might want to hide IDs depending on the type of data you're serving.

For example if your API involves sharing documents (public google docs) it's a bad idea to allow your user's files to be enumerable by simply incrementing the ID received from the API.

Or you could simply not want to allow others to see how many users or apps you have (or gain over time, if you're using ID offsets).

## How do we hide IDs?

For performance reasons you generally want the IDs to be reversible, therefore it's best to avoid directly hashing the ID. Instead you want to encrypt it before sending it to the user, and decrypt it when trying to access it in the backend.

Since the point of this post isn't security I'll be using the [hashids library](https://hashids.org). This gives us an easy way to generate unique strings from numbered IDs. Hashids is by no means secure, it's just a simple way to obfuscate IDs.

Here's how we use it in code:

```js
const hashids = new Hashids("secret salt");

const encodedId = hashids.encode(42);
const [originalId] = hashids.decode(encodedId);
```

## Express middleware

Let's assume that we want to hide **every** public ID. This would mean we encode/decode every `id` field in the request and response body. We could also expand this to relational fields, for example `user_id`. By doing the same to every field that ends in `_id`.

To achieve this in express, we want to create two middleware. `encodeMiddleware` and `decodeMiddleware`.

```js
/** helper function to replace IDs inside object */
function replaceIds(obj, replaceFunc) {
  if (obj == null) return obj;

  for (const key of Object.keys(obj)) {
    if (obj[key] == null) continue;

    if (typeof obj[key] === "object")
      obj[key] = replaceIds(obj[key], replaceFunc);
    else if (key == "id" || (key.length >= 4 && key.endsWith("_id")))
      obj[key] = replaceFunc(obj[key]);
  }
  return obj;
}

function encodeMiddleware(req, res, next) {
  var _json = res.json;
  res.json = (obj) => {
    res.json = _json;
    obj = replaceIds(obj, (v) => hashids.encode(v));
    return res.json(obj);
  };
  next();
}
function decodeMiddleware(req, res, next) {
  try {
    req.query = replaceIds(req.query, (v) => hashids.decode(v)[0]);
    req.body = replaceIds(req.body, (v) => hashids.decode(v)[0]);
  } catch (e) {
    console.error(`Could not decode id:`, e);
    return res.sendStatus(404);
  }
  next();
}
```

In the `encodeMiddleware` we're assuming the response will always be JSON and modifying it by replacing all `id` occurances with the encoded version.

In the `decodeMiddleware` we're assuming that the data can be either in the `body` or `query` and replacing all `id` occurances with the decoded version.

For both of these we could add exceptions to certain endpoints by comparing the `req.url`. Or just not using the middleware globaly.

I've also added a helper function that takes in an object and recursively replaces all `ids` using the supplied function.

## Example code

Now here's an example with everything put together:

```js
const express = require("express");
const Hashids = require("hashids");
const hashids = new Hashids("secret salt", 6);

/** helper function to recursively replace ids inside object */
function replaceIds(obj, replaceFunc) {
  if (obj == null) return obj;

  for (const key of Object.keys(obj)) {
    if (obj[key] == null) continue;

    if (typeof obj[key] === "object")
      obj[key] = replaceIds(obj[key], replaceFunc);
    else if (key == "id" || (key.length >= 4 && key.endsWith("_id")))
      obj[key] = replaceFunc(obj[key]);
  }
  return obj;
}

function encodeMiddleware(req, res, next) {
  var _json = res.json;
  res.json = (obj) => {
    res.json = _json;
    obj = replaceIds(obj, (v) => hashids.encode(v));
    return res.json(obj);
  };
  next();
}
function decodeMiddleware(req, res, next) {
  try {
    req.query = replaceIds(req.query, (v) => hashids.decode(v)[0]);
    req.body = replaceIds(req.body, (v) => hashids.decode(v)[0]);
  } catch (e) {
    console.error(`Could not decode id:`, e);
    return res.sendStatus(404);
  }
  next();
}

const app = express();
app.use(express.json());

// we're using the middleware globaly here
app.use(encodeMiddleware);
app.use(decodeMiddleware);

// sample endpoints to demonstrate encoding, decoding
app.get("/get-id", (req, res) => {
  res.json({ id: 5, name: "John" });
});
app.post("/send-id", (req, res) => {
  console.log(req.body);
  res.sendStatus(200);
});

app.listen(3000);
```

### Encoding and decoding

If we call the GET endpoint we should get back some JSON data with the `id` and `name`:

```bash
> curl GET http://localhost:3000/get-id

{"id":"OPZexb","name":"John"}%
```

Which we did, except the `id` was automatically encoded. If we were to return anything that ended with `_id` in the JSON body the middleware would automatically encode it.

Now let's try sending the encoded id to the POST endpoint:

```bash
> curl -X POST http://localhost:3000/send-id \
   -H 'Content-Type: application/json' \
   -d '{"id":"OPZexb"}'
...

[server log]: { id: 5 }
```

And on the server we should see `{ id: 5 }`. Which means the middleware successfully decoded the `id` we sent. Similarly if we send any values that might include `_id` it will automatically decode it.

## Closing notes

We were able to add global middleware whichs finds all `id`s in the request or response bodies and encodes or decodes them as needed.

In production you will likely want to add a filter to prevent this middleware running on webhooks from 3rd party services. As they might use the `id`, `_id` syntax themselves.


Add middleware that encodes/decodes every request to hide your ids

Hiding ids with express


Since this is a bit of a long post with multiple files I've provided a sample [repo](https://github.com/grawtar/express-validation-with-ts) so you can see the final code in action, in case the post gets a bit too hard to follow.

Most issues with Express crop up with badly (not) checked inputs to endpoints. The easiest way to show this is with an example:

```ts
type RequestBody<T> = Request<{}, {}, T>;

interface UserBody {
  name: string;
}
app.post("/user", (req: RequestBody<UserBody>, res) => {
  return addUser(req.body.name);
});
```

Here we're using typescript with express for [typed routes](https://urosstok.com/posts/typed-routes-in-express) which give us autocomplete.

Despite this being _typesafe_, it can still crash our whole backend!

Nothing is preventing someone from posting an empty body, which express will happily accept, which will cause an exception in the `addUser` function, which will crash the whole server as the error is not caught.

## Solution?

We could simply add some code checks (or catch some exceptions), for example:

```ts
type RequestBody<T> = Request<{}, {}, T>;

interface UserBody {
  name: string;
}
app.post("/user", (req: RequestBody<UserBody>, res) => {
  if (!req.body.name) return res.sendStatus(400); // we added this
  return addUser(req.body.name);
});
```

This is not a good solution though. For every new parameter we add to the endpoint (request body), we will have to make sure the code won't crash later. When you have a large API, it's really easy to miss some edge-case that brings the whole server down.

So what **can** you do?

## ajv

A library called [ajv](https://ajv.js.org/) offers _schema validation_, which allows you to validate data based on some pre-defined schema. This schema looks like this:

```ts
const schema = {
  type: "object",
  properties: {
    foo: { type: "integer" },
    bar: { type: "string", nullable: true },
  },
  required: ["foo"],
  additionalProperties: false,
};
```

Just as an example, if we were to describe this schema using a Typescript interface, we would get this:

```ts
interface UserPostRequest {
  foo: integer;
  foo: string;
}
```

But more on that later. First let's see how we could use _ajs_ to create a middleware that would only allow the request if the `req.body` matched the schema:

```ts
// function that creates middleware by compiling the supplied schema
function validateBody(schema) {
  // compile schema
  const validate = ajv.compile(schema);
  // middleware that returns error if schema is not ok
  return (req, res, next) => {
    if (!validate(req.body)) return res.status(400).json(validate.errors);
    return next();
  };
}

const schema = {
  type: "object",
  properties: {
    name: { type: "string" },
  },
  required: ["name"],
};

app.post("/user", validateBody(userSchema), (req, res) => {
  return addUser(req.body.name); // name will never be undefined
});
```

Now, wouldn't it be cool if we could use that typescript interface to... _generate_ the schema? What if we were able to convert our **typescript interface** into a **json schema**, which we would could then use for validation.

## typescript-json-schema

Now that cool thing we wanted to do? This [library](https://github.com/YousefED/typescript-json-schema) does exactly that!

It requires a bit of a setup.

- We need to have a `schema_definition.ts` file contains the interfaces we would like to convert into json schemas.
- Some `schemaGenerator.js` script that will use this library to convert said file into schema.
- And finally, a `_schema.ts` file that will be generated, all it will do is export the json schema so we can import it elsewhere. Essentialy a `.json` file wrapped in a `.ts` file.

To follow along easier you can clone the provided [repo](https://github.com/grawtar/express-validation-with-ts), which has this stuff already setup.

### schema_definition.ts

As mentioned, this will hold the **typescript interfaces** that will be converted into **json schemas**. For the user example let's just write:

```ts
// schema_definition.ts
export interface UserPostRequest {
  name: string;
}
```

### schemaGenerator.js

Pretty much the glue that holds this implementation together:

```js
// schemaGenerator.js
const path = require("path");
const tjs = require("typescript-json-schema");
const fs = require("fs");

const settings = {
  required: true,
  ref: false,
};
const compilerOptions = {
  strictNullChecks: true,
};

const program = tjs.getProgramFromFiles(
  [path.resolve("schema_definition.ts")],
  compilerOptions,
  "./"
);

const schema = tjs.generateSchema(program, "*", settings);
fs.writeFileSync(
  "_schema.ts",
  "const schema = " +
    JSON.stringify(schema) +
    " as const;\nexport default schema.definitions;"
);
```

And to use it, let's add a script to `package.json`:

```json
scripts: {
  "schema": "node schemaGenerator.js"
  ...
}
```

Running this with `npm run schema` will create a `_schema.ts` file in the root directory.

## Final code

If we were to expand our previous code, to support input validation, we would get this:

```ts
// index.ts
import express, { Request, Response, NextFunction } from "express";
import _schema from "./_schema";
import { UserPostRequest } from "./schema_definition";
import Ajv from "ajv";

const app = express();
app.use(express.json());

const ajv = new Ajv();

// validation middleware
function validateBody(schema: object) {
  const validate = ajv.compile(schema);
  return (req: any, res: any, next: NextFunction) => {
    if (!validate(req.body)) return res.status(400).json(validate.errors);
    return next();
  };
}

// helper type
type RequestBody<T> = Request<{}, {}, T>;

function addUser(name: string) {}

app.post(
  "/user",
  validateBody(_schema.UserPostRequest),
  (req: RequestBody<UserPostRequest>, res: Response) => {
    return addUser(req.body.name); // name will never be undefined
  }
);

app.listen(3000);
```

This whole example is available as a [repo](https://github.com/grawtar/express-validation-with-ts) if you're having issues setting things up.

## Closing notes

I hope you found this useful! Personally, it's my favourite way to add input validation if you're using typescript. It's so nice and clean!

Another cool alternative that allows for more powerful definitions (at the cost of losing clean interfaces) is [typebox](https://github.com/sinclairzx81/typebox).


Don't duplicate definitions by hand, you have Typescript

Input validation in Express from TypeScript


While Express wasn't built with Typescript, there are type definitions available - [@types/express](https://www.npmjs.com/package/@types/express). This adds typings for routes (specifically for this post, Request and Response).

I've looked around for ways of properly doing `Request` and `Response` types, and haven't found anything that works without breaking something else or being complicated. So here's how I usually implement typesafety into express routes.

Let's say we had an endpoint for adding a new user:

```ts
import express from "express";

const app = express();

app.post("/user", (req, res) => {
  req.body.name; // autocomplete doesn't work
});

app.listen(3000);
```

This is pretty standard javascript, besides using ESM imports, there's no reason we need typescript for this. So let's add some types:

```ts
import express, {Request, Response} from "express";
...
app.post("/user", (req: Request, res: Response) => {
    req.body.name; // autocomplete doesn't work
});
```

Note that this is what happens normally even if we don't specify the types, typescript infers the `Request` and `Response` type from the function automatically. So we didn't really do much here.

## Request.body type

What if this endpoint needs some input body data? Currently when we type `req.body` autocomplete doesn't offer anything special. Let's change that.

We can pass an interface to the `Request` type parameter list so that Typescript knows what variables are available in `req.body`. It would look something like this:

```ts
type UserRequestBody = { name: string };
app.post("/user", (req: Request<{}, {}, UserRequestBody>, res: Response) => {
  req.body.name; // autocomplete works
});
```

We need to put `{}` for the first two parameters as the thing we want (body) is actually the **third** type parameter. As we can see in the `Request` definition:

```ts
interface Request<
        P = core.ParamsDictionary,
        ResBody = any,
        ReqBody = any, // this is the Request.body
        ...
```

Now this is quite chunky code for simply passing an interface for the request body. Luckily there's a better way, we simply define a helper type:

```ts
type RequestBody<T> = Request<{}, {}, T>;
```

With our cleaner definition we can simply use:

```ts
type RequestBody<T> = Request<{}, {}, T>;

type UserRequestBody = { name: string };
app.post("/user", (req: RequestBody<UserRequestBody>, res: Response) => {
  req.body.name; // autocomplete works
});
```

## Other defintions

Now with our new found knowledge of how to write clean route typed code we can declare helper types for all our use cases!

```ts
// for .body
type RequestBody<T> = Request<{}, {}, T>;
// for .params
type RequestParams<T> = Request<T>;
// for .query
type RequestQuery<T> = Request<{}, {}, {}, T>;
// and so on... similarly for Response
```

## Multiple types

To cover everything, we need to be able to specify multiple types, for example `.body` and `.params`. We can do so by simply adding a new type:

```ts
type RequestBodyParams<TBody, TParams> = Request<TParams, {}, TBody>;
```

## Typed example

Here's the full example from the start, now with typed routes:

```ts
import express, { Request, Resposne } from "express";

const app = express();

type RequestBody<T> = Request<{}, {}, T>;
type UserRequestBody = { name: string };
app.post("/user", (req: RequestBody<UserRequestBody>, res: Response) => {
  req.body.name; // autocomplete works
});

app.listen(3000);
```

## Closing notes

That's it! This should allow you to create proper typed routes. The next step would be to add schema validation for these routes.


All posts