MUI Docs Infra

Warning

This is an internal project, and is not intended for public use. No support or stability guarantees are provided.

Load Code Variant

The loadCodeVariant module provides utilities for loading, processing, and transforming code variants with support for syntax highlighting, TypeScript-to-JavaScript transformation, extra files, and metadata management.

Note

This module is primarily used internally by CodeHighlighter for runtime code loading and processing. Most users won't need to call these functions directly.

Basic Usage

Loading a Code Variant

import { loadCodeVariant } from '@mui/internal-docs-infra/pipeline/loadCodeVariant';

// Load and process a single variant
const result = await loadCodeVariant(
  'file:///app/components/button/Button.tsx',
  'default',
  {
    fileName: 'Button.tsx',
    url: 'file:///app/components/button/Button.tsx',
  },
  {
    sourceParser: createParseSource(),
    loadSource: myLoadSourceFunction,
    sourceTransformers: [typescriptToJavaScript],
  },
);

// Returns:
// {
//   code: { fileName: 'Button.tsx', source: /* HAST nodes */, transforms: { ... } },
//   dependencies: ['file:///app/components/button/Button.tsx'],
//   externals: {}
// }

The function:

  • Loads source code from files or URLs
  • Applies syntax highlighting (HAST transformation)
  • Generates language transforms (e.g., TypeScript → JavaScript)
  • Recursively loads extra files (CSS, utils, etc.)
  • Tracks dependencies for caching and hot reloading

Common Patterns

Loading with Extra Files

Extra files can be specified directly in the variant, or returned automatically by your loadSource function:

// Option 1: Specify extraFiles in the variant
const result = await loadCodeVariant(
  'file:///app/components/button/Button.tsx',
  'default',
  {
    fileName: 'Button.tsx',
    source: buttonSource,
    extraFiles: {
      'Button.module.css': 'file:///app/components/button/Button.module.css',
      '../utils/helpers.ts': 'file:///app/components/utils/helpers.ts',
    },
  },
  options,
);

// Option 2: Return extraFiles from loadSource (automatically loaded)
const loadSource = async (url) => {
  const source = await readFile(url);
  const extraFiles = await findDependencies(url); // e.g., CSS imports, relative imports

  return {
    source,
    extraFiles: {
      'Button.module.css': 'file:///app/components/button/Button.module.css',
      '../utils/helpers.ts': 'file:///app/components/utils/helpers.ts',
    },
  };
};

// Both approaches result in all files loaded and highlighted recursively
// result.code.extraFiles contains processed CSS and helpers

Loading with Global Dependencies

Add shared dependencies (like package.json or global configs) to all variants:

const result = await loadCodeVariant(
  'file:///app/components/button/Button.tsx',
  'default',
  variant,
  {
    ...options,
    globalsCode: [
      { fileName: 'package.json', source: packageJsonContent },
      'file:///app/tsconfig.json',
    ],
  },
);

// Global files are merged with metadata flag for filtering

Parsing Code for Display

Convert code objects to HAST for rendering:

import { parseCode } from '@mui/internal-docs-infra/pipeline/loadCodeVariant';

const code = {
  default: {
    fileName: 'Example.tsx',
    source: 'const x = 1;', // String source
  },
};

const parsed = parseCode(code, parseSource);
// Converts string sources to HAST nodes for rendering

Enhancing Parsed Code

Use sourceEnhancers to modify HAST nodes after parsing. Enhancers receive the parsed HAST root, any comments extracted from the source (via loadServerSource), and the file name:

import type { SourceEnhancer, SourceComments } from '@mui/internal-docs-infra/CodeHighlighter';
import type { HastRoot } from 'hast';

// SourceComments is Record<number, string[]> - line numbers to comment arrays

// Example: Add line numbers based on comment sections
const addSectionMarkers: SourceEnhancer = (root, comments, fileName) => {
  // comments is Record<number, string[]> mapping line numbers to comment arrays
  // Use comments to enhance the HAST tree, e.g., add section markers
  if (comments) {
    for (const [lineNumber, lineComments] of Object.entries(comments)) {
      // Process comments to add markers, highlights, etc.
    }
  }
  return root;
};

// Example: Async enhancer that fetches additional data
const addTypeInfo: SourceEnhancer = async (root, comments, fileName) => {
  const typeData = await fetchTypeInfo(fileName);
  // Modify root with type information
  return root;
};

const result = await loadCodeVariant(
  'file:///app/components/button/Button.tsx',
  'default',
  variant,
  {
    sourceParser: createParseSource(),
    loadSource: myLoadSourceFunction,
    sourceEnhancers: [addSectionMarkers, addTypeInfo], // Run in order
  },
);

Enhancers run sequentially after parsing, each receiving the output of the previous enhancer. They are applied to both main files and extra files.

Flattening for File Systems

Convert nested variant structure to flat file paths:

import { flattenCodeVariant } from '@mui/internal-docs-infra/pipeline/loadCodeVariant';

const variant = {
  url: 'file:///src/components/button/Button.tsx',
  fileName: 'Button.tsx',
  source: buttonSource,
  extraFiles: {
    '../shared/utils.ts': { source: utilsSource },
    '../../package.json': { source: pkgSource, metadata: true },
  },
};

const flattened = flattenCodeVariant(variant);
// Returns:
// {
//   'src/components/button/Button.tsx': { source: buttonSource },
//   'src/components/shared/utils.ts': { source: utilsSource },
//   'src/package.json': { source: pkgSource, metadata: true }
// }

Checking Data Availability

Determine if you have enough data to render without loading:

import { maybeCodeInitialData } from '@mui/internal-docs-infra/pipeline/loadCodeVariant';

const { initialData, reason } = maybeCodeInitialData(
  ['typescript', 'javascript'],
  'typescript',
  code,
  undefined,
  true, // needsHighlight
  false, // needsAllFiles
  false, // needsAllVariants
);

if (initialData) {
  // Ready to render immediately
  return <CodeDisplay {...initialData} />;
} else {
  // Need to load more data
  console.log('Missing data:', reason);
}

Advanced Usage

Loading Fallback Data

Use loadCodeFallback to load the minimal data needed for initial render:

import { loadCodeFallback } from '@mui/internal-docs-infra/pipeline/loadCodeVariant';

const fallback = await loadCodeFallback(
  'file:///app/demos/example/index.ts',
  'default', // initialVariant
  existingCode, // may be undefined
  {
    loadCodeMeta,
    loadVariantMeta,
    loadSource,
    sourceParser,
    variants: ['default', 'typescript'],
    fallbackUsesExtraFiles: true,
    fallbackUsesAllVariants: false,
  },
);

// Returns minimal data for initial render:
// {
//   code: { default: { ... } },
//   initialFilename: 'Example.tsx',
//   initialSource: /* HAST nodes */,
//   allFileNames: ['Example.tsx', 'Example.module.css'],
// }

Applying Transforms

Apply TypeScript-to-JavaScript transforms to rendered code:

import { applyCodeTransform } from '@mui/internal-docs-infra/pipeline/loadCodeVariant';

const jsSource = applyCodeTransform(
  tsSource, // Original TypeScript HAST
  transforms, // From variant.transforms
  'javascript', // Transform key
);

// Returns JavaScript version of the source as HAST

Merging Metadata Files

Position metadata files (package.json, configs) relative to source files:

import { mergeCodeMetadata } from '@mui/internal-docs-infra/pipeline/loadCodeVariant';

const merged = mergeCodeMetadata(
  variant,
  {
    'package.json': { source: pkgJson },
    'tsconfig.json': { source: tsConfig },
  },
  { metadataPrefix: 'src/' },
);

// Positions metadata files at correct depth based on source structure

Examining Variant Structure

Analyze variant structure for path resolution:

import { examineCodeVariant } from '@mui/internal-docs-infra/pipeline/loadCodeVariant';

const context = examineCodeVariant(variant);
// Returns:
// {
//   hasUrl: true,
//   hasMetadata: true,
//   maxSourceBackNavigation: 2,
//   urlDirectory: ['src', 'components', 'button'],
//   rootLevel: 'src',
//   pathInwardFromRoot: 'components/button',
//   actualUrl: 'file:///src/components/button/Button.tsx'
// }

How It Works

Loading Pipeline

  1. Resolve Variant - If variant is a string URL, load metadata via loadVariantMeta
  2. Load Main File - Load source code via loadSource if not provided
  3. Apply Transforms - Generate language variants (TypeScript → JavaScript)
  4. Parse Source - Convert strings to HAST nodes via sourceParser
  5. Enhance HAST - Run sourceEnhancers sequentially with comments from source
  6. Process Extra Files - Recursively load and process dependencies
  7. Merge Globals - Add global dependencies with metadata flags
  8. Track Dependencies - Return all URLs for caching/hot reloading

Source Enhancement Pipeline

When sourceEnhancers are provided, they process the HAST after parsing:

// Enhancement flow
const enhancedRoot = await sourceEnhancers.reduce(async (rootPromise, enhancer) => {
  const root = await rootPromise;
  return enhancer(root, comments, fileName);
}, Promise.resolve(parsedRoot));

Each enhancer receives:

  • root - The current HAST root (output of previous enhancer)
  • comments - Comments extracted by loadSource (e.g., from parseImportsAndComments)
  • fileName - The file name for context

Enhancers can be synchronous or asynchronous and are applied to both main files and extra files.

Extra Files Resolution

Extra files support relative paths that are resolved recursively:

Button.tsx
├── Button.module.css (same directory)
├── ../utils/helpers.ts (parent directory)
└── ../../package.json (grandparent, metadata)

The loader:

  • Resolves relative paths using URL resolution
  • Detects circular dependencies
  • Loads files in parallel for performance
  • Preserves metadata flags for filtering

Transform Generation

When sourceTransformers are provided:

  1. Checks file extension (.ts, .tsx, etc.)
  2. Transforms source to target language
  3. Generates JSON diff delta
  4. Stores delta in transforms object

Transforms are applied on-demand during rendering via applyCodeTransform.

Metadata Positioning

The metadataPrefix option positions source files within a directory while keeping metadata files at the project root:

Without metadataPrefix:
├── Button.tsx (source)
├── utils.ts (source)
├── package.json (metadata)
└── tsconfig.json (metadata)

With metadataPrefix: 'src/':
├── src/
│   ├── Button.tsx (source)
│   └── utils.ts (source)
├── package.json (metadata - stays at root)
└── tsconfig.json (metadata - stays at root)

This ensures metadata files remain accessible at the project root while source files are scoped within their directory.


When to Use

  • Runtime loading - When code isn't precomputed at build time
  • Client-side rendering - Loading code in browser environments
  • Server-side rendering - Processing code in Next.js server components
  • Dynamic content - When demos are generated from APIs or databases

When NOT to use:

  • Build-time optimization - Use the webpack loader instead
  • Static demos - Pass precomputed code directly to CodeHighlighter
  • Simple highlighting - Use parseSource directly for basic needs

Performance Considerations

Parallel Loading

The loader uses Promise.all to load extra files in parallel:

// Loads all files simultaneously
extraFiles: {
  'styles.css': 'url1',
  'utils.ts': 'url2',
  'types.ts': 'url3',
}
// All three load in parallel, not sequentially

Caching

loadSource calls are cached within a single loadCodeVariant execution to prevent duplicate requests for the same file.

Circular Dependency Detection

The loader tracks loaded files and throws errors on circular dependencies:

// This would throw an error:
// A.tsx imports ../B.tsx
// B.tsx imports ../A.tsx

Output Formats

  • output: 'hastJson' - JSON stringified HAST (development)
  • output: 'hastGzip' - Compressed HAST (production, smaller bundles)

Error Handling

Common Errors

Circular dependency detected:

Error: Circular dependency detected: file:///path/to/file.ts

Fix: Restructure imports to remove circular references.

Invalid extraFiles:

Error: Invalid extraFiles from loadSource: key "file:///..." appears to be an absolute path.

Fix: Use relative paths as keys in extraFiles.

Missing loadSource:

Error: "loadSource" function is required when source is not provided

Fix: Provide loadSource in options or include source inline.

Transform not found:

Error: Transform "javascript" not found in transforms

Fix: Ensure sourceTransformers generated the requested transform.


Types

loadCodeVariant

loadCodeVariant

Loads a variant with support for recursive extra file loading. The loadSource function can now return extraFiles that will be loaded recursively. Supports both relative and absolute paths for extra files. Uses Promise.all for efficient parallel loading of extra files.

ParameterTypeDescription
urlstring | undefined

File URL for the variant

variantNamestring

Name of the variant (used for error messages)

variantVariantCode | string | undefined

Variant data object or URL string

optionsLoadVariantOptions | undefined

Loading and processing options (source parser, transformers, enhancers, etc.)

Return Type
Promise<{ code: VariantCode; dependencies: string[]; externals: Externals }>

loadCodeFallback

loadCodeFallback

Loads minimal data needed for fallback rendering. Returns code, initial filename, initial source, extra files, all file names, and processed globals code.

ParameterTypeDescription
urlstring

File URL for the variant

initialVariantstring

Name of the initial variant to load

loadedCode | undefined

Previously loaded Code object, if any

optionsLoadFallbackCodeOptions | undefined

Optional loading configuration

Return Type

parseCode

parseCode

Pure function to parse code variants and their extraFiles. Converts string sources to HAST nodes and handles hastJson parsing.

ParameterTypeDescription
code{ [key: string]: string | VariantCode | undefined }
parseSource((source: string, fileName: string, language?: string) => HastRoot)
Return Type
{ [key: string]: string | VariantCode | undefined }

flattenCodeVariant

flattenCodeVariant

Flatten a VariantCode into a flat files structure Resolves relative paths and handles metadata file scoping Uses addPathsToVariant for path resolution logic

ParameterTypeDescription
variantVariantCode
Return Type
{ [filePath: string]: FlatFile }

mergeCodeMetadata

mergeCodeMetadata
ParameterTypeDescription
variantVariantCode
metadataFilesVariantExtraFiles | undefined
optionsMergeMetadataOptions | undefined
Return Type
VariantCode

examineCodeVariant

examineCodeVariant

Create path context for processing files with extended information

ParameterTypeDescription
variantVariantCode
Return Type

maybeCodeInitialData

maybeCodeInitialData

Type guard function that determines if we have sufficient data to render a code highlighter component immediately, or if we need to start loading data first.

This function acts as a validation layer to ensure we have the minimal required data to render either a fallback state or the actual code content, helping to prevent rendering errors and provide better user experience.

Usage Contexts

This function is used in two main scenarios:

  1. Server-side rendering (CodeHighlighter): Determines if we can render with initial source content immediately, or if we need to load fallback data via CodeInitialSourceLoader

  2. Client-side hydration (CodeHighlighterClient): Within useInitialData hook to determine if we should trigger loading effects or if we can render with available data

Decision Flow

The function checks data availability in this order:

  1. Code object exists and contains the requested variant
  2. All required variants are available (if needsAllVariants is true)
  3. Requested file exists (main file or in extraFiles)
  4. All extra files are loaded (if needsAllFiles is true)
  5. Source content is properly highlighted (if needsHighlight is true)

Synchronous vs Asynchronous Behavior

This function operates synchronously and only validates existing data — it never triggers any loading operations. This design is crucial for performance and rendering strategies:

  • Synchronous validation allows immediate decisions about rendering paths without async overhead
  • Enables build-time optimization: When code is precomputed (e.g., via build-time processing), this function can immediately return initialData, avoiding async components entirely
  • Separates concerns: Data validation is separate from data loading, making the codebase more predictable and easier to reason about

When initialData: false is returned, the calling component is responsible for initiating asynchronous loading operations (e.g., loadCodeFallback, CodeInitialSourceLoader).

ParameterTypeDescription
variantsstring[]

Array of all available variant names for this code block (e.g., [‘javascript’, ‘typescript’])

variantstring

The specific variant we want to display (must exist in variants array)

codeCode | undefined

The code object containing all variant data (may be undefined if not loaded)

fileNamestring | undefined

Optional specific file to display. Resolution logic:

  • When it matches variantCode.fileName, uses the main variant source
  • When it doesn’t match, looks for the file in variantCode.extraFiles
  • When undefined, defaults to the main file of the variant
needsHighlightboolean | undefined

Whether the code needs to be syntax highlighted (source must be highlighted object, not string)

needsAllFilesboolean | undefined

Whether all extra files must be loaded before rendering (checks that all extraFiles have source content)

needsAllVariantsboolean | undefined

Whether all variants must be available before rendering (validates using hasAllVariants)

Return Type
KeyTypeDescription
initialData
| false
| {
    code: Code;
    initialFilename: string | undefined;
    initialSource: VariantSource;
    initialExtraFiles?: VariantExtraFiles;
  }
reasonstring | undefined

hasAllVariants

hasAllVariants

Determines if all code variants are fully loaded and ready to render the complete content component.

This function validates that we have all necessary data to transition from fallback/loading state to the full interactive code highlighter. It checks both main files and extra files for all variants.

Used primarily to determine when to show the full Content component instead of ContentLoading fallback, ensuring a smooth user experience without rendering errors.

ParameterTypeDescription
variantsstring[]

Array of variant names that must all be ready (e.g., [‘javascript’, ‘typescript’])

code{ [key: string]: string | VariantCode | undefined }

The code object containing variant data

needsHighlightboolean | undefined

Whether all sources need to be syntax highlighted (hast nodes, not strings)

Return Type
boolean

True if all variants and their files are loaded and ready for full rendering

enhanceCode

enhanceCode

Async function to enhance parsed code variants and their extraFiles. Applies sourceEnhancers to HAST nodes, using comments stored in the variant.

ParameterTypeDescription
code{ [key: string]: string | VariantCode | undefined }
sourceEnhancersSourceEnhancer[]
Return Type
Promise<Code>

applyCodeTransform

applyCodeTransform

Applies a specific transform to a variant source and returns the transformed source

ParameterTypeDescription
sourceVariantSource

The original variant source (string, HastNodes, or hastJson object)

transforms{ [key: string]: { delta: Delta; fileName?: string } }

Object containing all available transforms

transformKeystring

The key of the specific transform to apply

Return Type
VariantSource

The transformed variant source in the same format as the input

addPathsToVariant

addPathsToVariant

Add flat paths to all files in a variant

ParameterTypeDescription
variantVariantCode
Return Type
VariantCode

Additional Types

FallbackVariants
type FallbackVariants = {
  code: Code;
  initialFilename: string | undefined;
  initialSource: VariantSource;
  initialExtraFiles?: VariantExtraFiles;
  allFileNames: string[];
  processedGlobalsCode?: Code[];
};
FileWithPath
type FileWithPath = { source?: VariantSource; metadata?: boolean; path: string };
FlatFile
type FlatFile = { source: string; metadata?: boolean };
FlattenedFiles
type FlattenedFiles = { [filePath: string]: FlatFile };
PathContext
type PathContext = PathContextWithUrl | PathContextWithoutUrl;

Related