Managing robots.txt and Root Text Files in Optimizely CMS - Introducing Virtual Text

In many Optimizely CMS projects, certain files must exist at the root of the site - robots.txt, ads.txt, security.txt, or other plain text endpoints required by search engines, security standards, or integrations.

These files are often managed outside the CMS and require deployments for even small changes. That workflow is inconvenient for editors and adds unnecessary friction for operations.

To address this, I developed a small open-source plugin called Virtual Text, which allows managing and rendering root-level text files directly from the Optimizely back office.

Repository:
https://github.com/ddprince17/davidhome-optimizely-virtualtext

What Virtual Text Does

Virtual Text extends Optimizely CMS with the ability to define and serve text files directly from the application pipeline rather than from physical files on disk.

Editors can manage files such as:

• robots.txt
• ads.txt
• .well-known/security.txt
• Any other root-level text file

All content is editable through the CMS interface and served dynamically by middleware.

Multi-Site and Hostname Awareness

The plugin was built with real Optimizely environments in mind, including multi-site setups.

Virtual Text supports:

• Per-site configuration
• Per-site hostname configuration

This allows different sites or hostnames within the same CMS instance to expose different text files or content when required.

Storage and Architecture

The plugin separates file locations from file contents and supports Azure-backed storage providers.

Available packages include:

Core:

• VirtualText core plugin
• Azure Table provider for file locations
• Azure Blob provider for file contents

This approach allows scaling storage independently and avoids coupling text files to local disk.

RobotsTxt Extension for Non-Production Environments

The repository also includes an optional RobotsTxt extension designed to solve a common operational problem - preventing indexing of test and staging environments.

Features include:

• Environment-based indexing policies
• Automatic robots.txt manipulation in non-production environments
• Default directives such as disallowing all crawling
• Optional emission of indexing directives

This reduces the risk of staging or QA environments being indexed accidentally, which is still a surprisingly common issue in enterprise projects.

Permissions and Governance

Virtual Text integrates with Optimizely permissions to control access:

• View permissions
• Edit permissions

This allows delegating responsibility safely to operations or marketing teams without exposing broader CMS administration rights.

Why I Built This

In multiple projects, I repeatedly encountered the same friction:

• Editors needing to update robots.txt or ads.txt
• Developers being asked to deploy small file changes
• Risk of staging environments being indexed
• Multi-site environments requiring different configurations

Virtual Text was built to solve these problems in a reusable and consistent way.

Getting Started

Installation is straightforward:

• Add the NuGet packages
• Register the services
• Configure optional settings

Full instructions and examples are available in the repository:
https://github.com/ddprince17/davidhome-optimizely-virtualtext

Feedback, issues, and contributions are always welcome!