unified

Project: unifiedjs/unified

Package: unified@10.1.2

  1. Dependents: 0
  2. Interface for parsing, inspecting, transforming, and serializing content through syntax trees
  1. remark 193
  2. unified 182
  3. rehype 91
  4. tree 44
  5. retext 41
  6. ast 37
  7. syntax 28
  8. parse 24
  9. content 20
  10. stringify 19
  11. compile 14
  12. serialize 12
  13. process 7
  14. transform 7
  15. cst 6

unified

Build Coverage Downloads Size Sponsors Backers Chat

unified lets you inspect and transform content with plugins.

Contents

What is this?

unified is two things:

Several ecosystems are built on unified around different kinds of content. Notably, remark (markdown), rehype (HTML), and retext (natural language). These ecosystems can be connected together.

When should I use this?

In some cases, you are already using unified. For example, it’s used in MDX, Gatsby, Docusaurus, etc. In those cases, you don’t need to add unified yourself but you can include plugins into those projects.

But the real fun (for some) is to get your hands dirty and work with syntax trees and build with it yourself. You can create those projects, or things like Prettier, or your own site generator. You can connect utilities together and make your own plugins that check for problems and transform from one thing to another.

When you are dealing with one type of content (such as markdown), it’s recommended to use the main package of that ecosystem instead (so remark). When you are dealing with different kinds of content (such as markdown and HTML), it’s recommended to use unified itself, and pick and choose the plugins you need.

Install

This package is ESM only. In Node.js (version 12.20+, 14.14+, or 16.0+), install with npm:

npm install unified

In Deno with esm.sh:

import {unified} from 'https://esm.sh/unified@10'

In browsers with esm.sh:

<script type="module">
  import {unified} from 'https://esm.sh/unified@10?bundle'
</script>

Use

import {unified} from 'unified'
import remarkParse from 'remark-parse'
import remarkRehype from 'remark-rehype'
import rehypeDocument from 'rehype-document'
import rehypeFormat from 'rehype-format'
import rehypeStringify from 'rehype-stringify'
import {reporter} from 'vfile-reporter'

const file = await unified()
  .use(remarkParse)
  .use(remarkRehype)
  .use(rehypeDocument, {title: '👋🌍'})
  .use(rehypeFormat)
  .use(rehypeStringify)
  .process('# Hello world!')

console.error(reporter(file))
console.log(String(file))

Yields:

no issues found
<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <title>👋🌍</title>
    <meta name="viewport" content="width=device-width, initial-scale=1">
  </head>
  <body>
    <h1>Hello world!</h1>
  </body>
</html>

Overview

unified is an interface for processing content with syntax trees. Syntax trees are a representation of content understandable to programs. Those programs, called plugins, take these trees and inspect and modify them. To get to the syntax tree from text, there is a parser. To get from that back to text, there is a compiler. This is the process of a processor.

| ........................ process ........................... |
| .......... parse ... | ... run ... | ... stringify ..........|

          +--------+                     +----------+
Input ->- | Parser | ->- Syntax Tree ->- | Compiler | ->- Output
          +--------+          |          +----------+
                              X
                              |
                       +--------------+
                       | Transformers |
                       +--------------+
Processors

Processors process content. On its own, unified (the root processor) doesn’t work. It needs to be configured with plugins to work. For example:

const processor = unified()
  .use(remarkParse)
  .use(remarkRehype)
  .use(rehypeDocument, {title: '👋🌍'})
  .use(rehypeFormat)
  .use(rehypeStringify)

That processor can do different things. It can:

Every processor implements another processor. To create a processor, call another processor. The new processor is configured to work the same as its ancestor. But when the descendant processor is configured in the future it does not affect the ancestral processor.

When processors are exposed from a module (for example, unified itself) they should not be configured directly, as that would change their behavior for all module users. Those processors are frozen and they should be called to create a new processor before they are used.

File

When processing a document, metadata is gathered about that document. vfile is the file format that stores data, metadata, and messages about files for unified and plugins.

There are several utilities for working with these files.

Syntax tree

The syntax trees used in unified are unist nodes. A tree represents a whole document and each node is a plain JavaScript object with a type field. The semantics of nodes and the format of syntax trees is defined by other projects:

There are many utilities for working with trees listed in each aforementioned project and maintained in the syntax-tree organization. These utilities are a level lower than unified itself and are building blocks that can be used to make plugins.

Ecosystems

Around each syntax tree is an ecosystem that focusses on that particular kind of content. At their core, they parse text to a tree and compile that tree back to text. They also provide plugins that work with the syntax tree, without requiring that the end user has knowledge about that tree.

Plugins

Each aforementioned ecosystem comes with a large set of plugins that you can pick and choose from to do all kinds of things.

There are also a few plugins that work in any ecosystem:

Configuration

Processors are configured with plugins or with the data method. Most plugins also accept configuration through options. See each plugin’s readme for more info.

Integrations

unified can integrate with the file system through unified-engine. CLI apps can be created with unified-args, Gulp plugins with unified-engine-gulp, and language servers with unified-language-server. A streaming interface can be created with unified-stream.

Programming interface

The API provided by unified allows multiple files to be processed and gives access to metadata (such as lint messages):

import {unified} from 'unified'
import remarkParse from 'remark-parse'
import remarkPresetLintMarkdownStyleGuide from 'remark-preset-lint-markdown-style-guide'
import remarkRetext from 'remark-retext'
import retextEnglish from 'retext-english'
import retextEquality from 'retext-equality'
import remarkRehype from 'remark-rehype'
import rehypeStringify from 'rehype-stringify'
import {reporter} from 'vfile-reporter'

const file = await unified()
  .use(remarkParse)
  .use(remarkPresetLintMarkdownStyleGuide)
  .use(remarkRetext, unified().use(retextEnglish).use(retextEquality))
  .use(remarkRehype)
  .use(rehypeStringify)
  .process('*Emphasis* and _stress_, you guys!')

console.error(reporter(file))
console.log(String(file))

Yields:

  1:16-1:24  warning  Emphasis should use `*` as a marker                                  emphasis-marker  remark-lint
  1:30-1:34  warning  `guys` may be insensitive, use `people`, `persons`, `folks` instead  gals-man         retext-equality

⚠ 2 warnings
<p><em>Emphasis</em> and <em>stress</em>, you guys!</p>

Transforming between ecosystems

Ecosystems can be combined in two modes.

Bridge mode transforms the tree from one format (origin) to another (destination). A different processor runs on the destination tree. Afterwards, the original processor continues with the origin tree.

Mutate mode also transforms the syntax tree from one format to another. But the original processor continues transforming the destination tree.

In the previous example (“Programming interface”), remark-retext is used in bridge mode: the origin syntax tree is kept after retext is done; whereas remark-rehype is used in mutate mode: it sets a new syntax tree and discards the origin tree.

The following plugins lets you combine ecosystems:

API

This package exports the identifier unified (the root processor). There is no default export.

processor()

Create a processor.

Returns

New unfrozen processor (processor) that is configured to work the same as its ancestor. When the descendant processor is configured in the future it does not affect the ancestral processor.

Example

This example shows how a new processor can be created (from remark) and linked to stdin(4) and stdout(4).

import process from 'node:process'
import concatStream from 'concat-stream'
import {remark} from 'remark'

process.stdin.pipe(
  concatStream((buf) => {
    process.stdout.write(String(remark().processSync(buf)))
  })
)

processor.use(plugin[, options])

Configure the processor to use a plugin and optionally configure that plugin with options.

If the processor is already using a plugin, the previous plugin configuration is changed based on the options that are passed in. In other words, the plugin is not added a second time.

👉 Note: use cannot be called on frozen processors. Call the processor first to create a new unfrozen processor.

Signatures
Parameters
Returns

The processor that use was called on (processor).

Example

There are many ways to pass plugins to .use(). This example gives an overview:

import {unified} from 'unified'

unified()
  // Plugin with options:
  .use(pluginA, {x: true, y: true})
  // Passing the same plugin again merges configuration (to `{x: true, y: false, z: true}`):
  .use(pluginA, {y: false, z: true})
  // Plugins:
  .use([pluginB, pluginC])
  // Two plugins, the second with options:
  .use([pluginD, [pluginE, {}]])
  // Preset with plugins and settings:
  .use({plugins: [pluginF, [pluginG, {}]], settings: {position: false}})
  // Settings only:
  .use({settings: {position: false}})

processor.parse(file)

Parse text to a syntax tree.

👉 Note: parse freezes the processor if not already frozen.

👉 Note: parse performs the parse phase, not the run phase or other phases.

Parameters
Returns

Syntax tree representing file (Node).

Example

This example shows how parse can be used to create a tree from a file.

import {unified} from 'unified'
import remarkParse from 'remark-parse'

const tree = unified().use(remarkParse).parse('# Hello world!')

console.log(tree)

Yields:

{
  type: 'root',
  children: [
    {type: 'heading', depth: 1, children: [Array], position: [Position]}
  ],
  position: {
    start: {line: 1, column: 1, offset: 0},
    end: {line: 1, column: 15, offset: 14}
  }
}

processor.Parser

A parser handles the parsing of text to a syntax tree. It is used in the parse phase and is called with a string and VFile of the document to parse.

Parser can be a normal function, in which case it must return the syntax tree representation of the given file (Node).

Parser can also be a constructor function (a function with a parse field, or other fields, in its prototype), in which case it is constructed with new. Instances must have a parse method that is called without arguments and must return a Node.

processor.stringify(tree[, file])

Compile a syntax tree.

👉 Note: stringify freezes the processor if not already frozen.

👉 Note: stringify performs the stringify phase, not the run phase or other phases.

Parameters
Returns

Textual representation of the tree (string or Buffer, see note).

👉 Note: unified typically compiles by serializing: most compilers return string (or Buffer). Some compilers, such as the one configured with rehype-react, return other values (in this case, a React tree). If you’re using a compiler that doesn’t serialize, expect different result values.

Example

This example shows how stringify can be used to serialize a syntax tree:

import {unified} from 'unified'
import rehypeStringify from 'rehype-stringify'
import {h} from 'hastscript'

const tree = h('h1', 'Hello world!')

const doc = unified().use(rehypeStringify).stringify(tree)

console.log(doc)

Yields:

<h1>Hello world!</h1>

processor.Compiler

A compiler handles the compiling of a syntax tree to something else (in most cases, text). It is used in the stringify phase and called with a Node and VFile representation of the document to compile.

Compiler can be a normal function, in which case it should return the textual representation of the given tree (string).

Compiler can also be a constructor function (a function with a compile field, or other fields, in its prototype), in which case it is constructed with new. Instances must have a compile method that is called without arguments and should return a string.

👉 Note: unified typically compiles by serializing: most compilers return string (or Buffer). Some compilers, such as the one configured with rehype-react, return other values (in this case, a React tree). If you’re using a compiler that doesn’t serialize, expect different result values.

processor.run(tree[, file][, done])

Run transformers on a syntax tree.

👉 Note: run freezes the processor if not already frozen.

👉 Note: run performs the run phase, not other phases.

Parameters
Returns

Nothing if done is given (void). A Promise otherwise. The promise is rejected with a fatal error or resolved with the transformed tree (Node).

Example

This example shows how run can be used to transform a tree:

import {unified} from 'unified'
import remarkReferenceLinks from 'remark-reference-links'
import {u} from 'unist-builder'

const tree = u('root', [
  u('paragraph', [
    u('link', {href: 'https://example.com'}, [u('text', 'Example Domain')])
  ])
])

const changedTree = await unified().use(remarkReferenceLinks).run(tree)

console.log(changedTree)

Yields:

{
  type: 'root',
  children: [
    {type: 'paragraph', children: [Array]},
    {type: 'definition', identifier: '1', title: undefined, url: undefined}
  ]
}

function done(err[, tree, file])

Callback called when transformers are done. Called with either an error or results.

Parameters

processor.runSync(tree[, file])

Run transformers on a syntax tree. An error is thrown if asynchronous transforms are configured.

👉 Note: runSync freezes the processor if not already frozen.

👉 Note: runSync performs the run phase, not other phases.

Parameters
Returns

Transformed tree (Node).

processor.process(file[, done])

Process the given file as configured on the processor.

👉 Note: process freezes the processor if not already frozen.

👉 Note: process performs the parse, run, and stringify phases.

Parameters
Returns

Nothing if done is given (void). A Promise otherwise. The promise is rejected with a fatal error or resolved with the processed file (VFile).

The parsed, transformed, and compiled value is available at file.value (see note).

👉 Note: unified typically compiles by serializing: most compilers return string (or Buffer). Some compilers, such as the one configured with rehype-react, result in other values (in this case, a React tree). If you’re using a compiler that does not serialize, the result is available at file.result.

Example

This example shows how process can be used to process a file:

import {unified} from 'unified'
import remarkParse from 'remark-parse'
import remarkRehype from 'remark-rehype'
import rehypeDocument from 'rehype-document'
import rehypeFormat from 'rehype-format'
import rehypeStringify from 'rehype-stringify'

const file = await unified()
  .use(remarkParse)
  .use(remarkRehype)
  .use(rehypeDocument, {title: '👋🌍'})
  .use(rehypeFormat)
  .use(rehypeStringify)
  .process('# Hello world!')

console.log(String(file))

Yields:

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <title>👋🌍</title>
    <meta name="viewport" content="width=device-width, initial-scale=1">
  </head>
  <body>
    <h1>Hello world!</h1>
  </body>
</html>

function done(err, file)

Callback called when the process is done. Called with either an error or a result.

Parameters
Example

This example shows how process can be used to process a file with a callback.

import {unified} from 'unified'
import remarkParse from 'remark-parse'
import remarkGithub from 'remark-github'
import remarkStringify from 'remark-stringify'
import {reporter} from 'vfile-reporter'

unified()
  .use(remarkParse)
  .use(remarkGithub)
  .use(remarkStringify)
  .process('@unifiedjs', function (error, file) {
    console.error(reporter(error || file))
    if (file) {
      console.log(String(file))
    }
  })

Yields:

no issues found
[**@unifiedjs**](https://github.com/unifiedjs)

processor.processSync(file)

Process the given file as configured on the processor. An error is thrown if asynchronous transforms are configured.

👉 Note: processSync freezes the processor if not already frozen.

👉 Note: processSync performs the parse, run, and stringify phases.

Parameters
Returns

The processed file (VFile).

The parsed, transformed, and compiled value is available at file.value (see note).

👉 Note: unified typically compiles by serializing: most compilers return string (or Buffer). Some compilers, such as the one configured with rehype-react, result in other values (in this case, a React tree). If you’re using a compiler that does not serialize, the result is available at file.result.

Example

This example shows how processSync can be used to process a file, if all transformers are synchronous.

import {unified} from 'unified'
import remarkParse from 'remark-parse'
import remarkRehype from 'remark-rehype'
import rehypeDocument from 'rehype-document'
import rehypeFormat from 'rehype-format'
import rehypeStringify from 'rehype-stringify'

const processor = unified()
  .use(remarkParse)
  .use(remarkRehype)
  .use(rehypeDocument, {title: '👋🌍'})
  .use(rehypeFormat)
  .use(rehypeStringify)

console.log(String(processor.processSync('# Hello world!')))

Yields:

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <title>👋🌍</title>
    <meta name="viewport" content="width=device-width, initial-scale=1">
  </head>
  <body>
    <h1>Hello world!</h1>
  </body>
</html>

processor.data([key[, value]])

Configure the processor with info available to all plugins. Information is stored in an object.

Typically, options can be given to a specific plugin, but sometimes it makes sense to have information shared with several plugins. For example, a list of HTML elements that are self-closing, which is needed during all phases.

👉 Note: setting information cannot occur on frozen processors. Call the processor first to create a new unfrozen processor.

Signatures
Parameters
Returns
Example

This example show how to get and set info:

import {unified} from 'unified'

const processor = unified().data('alpha', 'bravo')

processor.data('alpha') // => 'bravo'

processor.data() // => {alpha: 'bravo'}

processor.data({charlie: 'delta'})

processor.data() // => {charlie: 'delta'}

processor.freeze()

Freeze a processor. Frozen processors are meant to be extended and not to be configured directly.

When a processor is frozen it cannot be unfrozen. New processors working the same way can be created by calling the processor.

It’s possible to freeze processors explicitly by calling .freeze(). Processors freeze automatically when .parse(), .run(), .runSync(), .stringify(), .process(), or .processSync() are called.

Returns

The processor that freeze was called on (processor).

Example

This example, index.js, shows how rehype prevents extensions to itself:

import {unified} from 'unified'
import rehypeParse from 'rehype-parse'
import rehypeStringify from 'rehype-stringify'

export const rehype = unified().use(rehypeParse).use(rehypeStringify).freeze()

That processor can be used and configured like so:

import {rehype} from 'rehype'
import rehypeFormat from 'rehype-format'
// …

rehype()
  .use(rehypeFormat)
  // …

A similar looking example is broken as operates on the frozen interface. If this behavior was allowed it would result in unexpected behavior so an error is thrown. This is not valid:

import {rehype} from 'rehype'
import rehypeFormat from 'rehype-format'
// …

rehype
  .use(rehypeFormat)
  // …

Yields:

~/node_modules/unified/index.js:426
    throw new Error(
    ^

Error: Cannot call `use` on a frozen processor.
Create a new processor first, by calling it: use `processor()` instead of `processor`.
    at assertUnfrozen (~/node_modules/unified/index.js:426:11)
    at Function.use (~/node_modules/unified/index.js:165:5)
    …

Plugin

Plugins configure the processors they are applied on in the following ways:

Plugins are a concept. They materialize as Attachers.

Example

move.js:

/**
 * @typedef Options
 *   Configuration (required).
 * @property {string} extname
 *   File extension to use (must start with `.`).
 */

/** @type {import('unified').Plugin<[Options]>} */
export function move(options) {
  if (!options || !options.extname) {
    throw new Error('Missing `options.extname`')
  }

  return function (tree, file) {
    if (file.extname && file.extname !== options.extname) {
      file.extname = options.extname
    }
  }
}

index.md:

# Hello, world!

index.js:

import {read, write} from 'to-vfile'
import {reporter} from 'vfile-reporter'
import {unified} from 'unified'
import remarkParse from 'remark-parse'
import remarkRehype from 'remark-rehype'
import rehypeStringify from 'rehype-stringify'
import {move} from './move.js'

const file = await unified()
  .use(remarkParse)
  .use(remarkRehype)
  .use(move, {extname: '.html'})
  .use(rehypeStringify)
  .process(await read('index.md'))

console.error(reporter(file))
await write(file) // Written to `index.html`.

Yields:

index.md: no issues found

…and in index.html:

<h1>Hello, world!</h1>

function attacher(options?)

Attachers are materialized plugins. They are functions that can receive options and configure the processor.

Attachers change the processor, such as the parser, the compiler, by configuring data, or by specifying how the tree and file are handled.

👉 Note: attachers are called when the processor is frozen, not when they are applied.

Parameters
Returns

Optional transform (Transformer).

function transformer(tree, file[, next])

Transformers handle syntax trees and files. They are functions that are called each time a syntax tree and file are passed through the run phase. When an error occurs in them (either because it’s thrown, returned, rejected, or passed to next), the process stops.

The run phase is handled by trough, see its documentation for the exact semantics of these functions.

Parameters
Returns

function next(err[, tree[, file]])

If the signature of a transformer accepts a third argument, the transformer may perform asynchronous operations, and must call next().

Parameters

Preset

Presets are sharable configuration. They can contain plugins and settings.

Example

preset.js:

import remarkPresetLintRecommended from 'remark-preset-lint-recommended'
import remarkPresetLintConsistent from 'remark-preset-lint-consistent'
import remarkCommentConfig from 'remark-comment-config'
import remarkToc from 'remark-toc'
import remarkLicense from 'remark-license'

export const preset = {
  settings: {bullet: '*', emphasis: '*', fences: true},
  plugins: [
    remarkPresetLintRecommended,
    remarkPresetLintConsistent,
    remarkCommentConfig,
    [remarkToc, {maxDepth: 3, tight: true}],
    remarkLicense
  ]
}

example.md:

# Hello, world!

_Emphasis_ and **importance**.

## Table of contents

## API

## License

index.js:

import {read, write} from 'to-vfile'
import {remark} from 'remark'
import {reporter} from 'vfile-reporter'
import {preset} from './preset.js'

const file = await remark()
  .use(preset)
  .process(await read('example.md'))

console.error(reporter(file))
await write(file)

Yields:

example.md: no issues found

example.md now contains:

# Hello, world!

*Emphasis* and **importance**.

## Table of contents

*   [API](#api)
*   [License](#license)

## API

## License

[MIT](license) © [Titus Wormer](https://wooorm.com)

Types

This package is fully typed with TypeScript. It exports the following additional types:

For TypeScript to work, it is particularly important to type your plugins correctly. We strongly recommend using the Plugin type with its generics and to use the node types for the syntax trees provided by our packages (as in, @types/hast, @types/mdast, @types/nlcst).

/**
 * @typedef {import('mdast').Root} MdastRoot
 * @typedef {import('hast').Root} HastRoot
 *
 * @typedef Options
 *   Configuration (optional).
 * @property {boolean} [someField]
 *   Some option.
 */

// To type options:
/** @type {import('unified').Plugin<[Options?]>} */
export function myPluginAcceptingOptions(options) {
  // `options` is `Options?`.
}

// To type a plugin that works on a certain tree:
/** @type {import('unified').Plugin<[], MdastRoot>} */
export function myRemarkPlugin() {
  return function (tree, file) {
    // `tree` is `MdastRoot`.
  }
}

// To type a plugin that transforms one tree into another:
/** @type {import('unified').Plugin<[], MdastRoot, HastRoot>} */
export function remarkRehype() {
  return function (tree) {
    // `tree` is `MdastRoot`.
    // Result must be `HastRoot`.
  }
}

// To type a plugin that defines a parser:
/** @type {import('unified').Plugin<[], string, MdastRoot>} */
export function remarkParse(options) {}

// To type a plugin that defines a compiler:
/** @type {import('unified').Plugin<[], HastRoot, string>} */
export function rehypeStringify(options) {}

Compatibility

Projects maintained by the unified collective are compatible with all maintained versions of Node.js. As of now, that is Node.js 12.20+, 14.14+, and 16.0+. Our projects sometimes work with older versions, but this is not guaranteed.

Contribute

See contributing.md in unifiedjs/.github for ways to get started. See support.md for ways to get help.

This project has a code of conduct. By interacting with this repository, organization, or community you agree to abide by its terms.

For info on how to submit a security report, see our security policy.

Support this effort and give back by sponsoring on OpenCollective!

Vercel

Motif

HashiCorp

American Express

GitBook

Gatsby

Netlify

Coinbase

ThemeIsle

Expo

Boost Note

Holloway


You?

Acknowledgments

Preliminary work for unified was done in 2014 for retext and inspired by ware. Further incubation happened in remark. The project was finally externalised in 2015 and published as unified. The project was authored by @wooorm.

Although unified since moved its plugin architecture to trough, thanks to @calvinfo, @ianstormtaylor, and others for their work on ware, as it was a huge initial inspiration.

License

MIT © Titus Wormer