Nuclei YAML Syntax Documentation
A common problem with YAML syntax is figuring out all the supported fields and their relevant documentation. With Nuclei, this has always been a barrier of entry for people new to the project.
This Automatically Generated Documentation for Nuclei YAML Syntax is a step in this direction, making the Nuclei Syntax easier for people to understand and get working with. The documentation is generated using yamldoc-go, which is based on the amazing work of talos-system people.
A link to the documentation is provided below. It contains all the fields in the syntax, along with descriptions and examples.
projectdiscoveryThe documentation is generated from code on each release, so you can be sure that it will always be up-to-date with the latest engine features.
Environment & CLI Variables Support
This addition allows using values from Environment Variable inside a template with an optional flag -env-vars. When enabled, all the values in env are available for use in a template.
requests:
- method: GET
path:
- "{{BaseURL}}"
headers:
Content-Type: '{{ENV_TEST}}'Template-based Environment variable support
In the above example, the ENV_TEST string will be replaced with the environment variable value of ENV_TEST.
It has also been made possible to specify these variables from CLI. This can be done using the -var key=value flag syntax. The above example ENV_TEST can be replicated from CLI with the -var ENV_TEST=example flag.
This feature also allows us to create templates that conduct authenticated actions without hardcoding keys or credentials.
JSON and XPath Extractors
Two new extractor types have been added to nuclei - json and xpath.
JSON extractor allows using JQ-style syntax to extract items from json responses. An example of using JSON extractor to extract IDs from Gitlab project ids is provided below.
extractors:
- type: json
part: body
json:
- '.[] | .id'JSON extractor to extract Gitlab project ids
XPath extractor allows using XPath expressions to extract items from HTML responses. An optional attribute to extract can also be provided.
extractors:
- type: xpath
part: body
attribute: href
xpath:
- "/html/body/div/p[2]/a"
Extracting a link using XPath extractor
Elasticsearch Exporter
It is now possible to export results directly from nuclei to Elasticsearch. This can be used for result visualization as well as data storage. The below-provided configuration file can be used with nuclei to export data to Elasticsearch.
elasticsearch:
# IP for elasticsearch instance
ip: 127.0.0.1
# Port is the port of elasticsearch instance
port: 9200
# IndexName is the name of the elasticsearch index
index-name: nuclei
# SSL enables ssl for elasticsearch connection
# ssl: false
# SSLVerification disables SSL verification for elasticsearch
# ssl-verification: false
# Username for the elasticsearch instance
# username: test
# Pasword is the password for elasticsearch instance
# password: test Elasticsearch Reporting Config Example
Running nuclei with nuclei -rc config.yaml -t <templates> -l <list> sends the found results to Elasticsearch, which can now be visualized with Kibana and other solutions.
JSONSchema Support for Nuclei Syntax
An automatically generated json-schema for the Nuclei YAML Syntax has been added to the repository as well. The link is provided below -
projectdiscoveryThis schema can be used as a starting point for Nuclei Intellisense support. A more detailed Blog Post regarding all Editor integrations will come soon; however, until then, you can follow the medium post here to set up Nuclei YAML JSONSchema in your editor. The link provided is for VSCode; however, it should be similar for other editors.
More features for HTTP Protocol
New Global Variables
A list of new variables has been added to Nuclei HTTP Module, generated from the provided URL, and can be used anywhere within the request block. An example is provided below, including new variables -
https://example.com:443/foo/bar.php
| Variable | Value |
|---|---|
| {{BaseURL}} | https://example.com:443/foo/bar.php |
| {{RootURL}} | https://example.com:443 |
| {{Hostname}} | example.com:443 |
| {{Host}} | example.com |
| {{Port}} | 443 |
| {{Path}} | /foo |
| {{File}} | bar.php |
| {{Scheme}} | https |
Unified Attack mode support
The nuclei Attack module where the engine tries a combination of payload values and helper functions is now available for use in the Base HTTP request. Earlier, these could only be used in conjunction with Raw HTTP requests.
id: basic-http-payload-helpers-example
info:
name: Test HTTP Payload Template
author: pdteam
severity: info
requests:
- method: GET
path:
- "{{BaseURL}}/{{path}}"
headers:
hello: "{{md5('Hello')}}"
payloads:
path:
- abc
- dsa
attack: sniper
matchers:
- type: status
status:
- 200Basic HTTP Payload + Helper Function Example
Stop At First Match
stop-at-first-match is now supported in the template for HTTP requests. What this means is, if there are multiple requests, nuclei will stop sending requests as soon as it gets a match. This is useful in cases like brute-forcing, etc., where you want to stop after finding the first match. Previously, it was supported by a CLI flag that applied to all templates, and now it can be defined within a template, allowing it to be used in particular cases.
id: test-stop-at-first-match
info:
name: test-stop-at-first-match
author: pdteam
severity: info
requests:
- method: GET
path:
- "{{BaseURL}}/php.php"
- "{{BaseURL}}/phpinfo.php"
- "{{BaseURL}}/info.php"
- "{{BaseURL}}/infophp.php"
- "{{BaseURL}}/php_info.php"
- "{{BaseURL}}/test.php"
- "{{BaseURL}}/i.php"
- "{{BaseURL}}/asdf.php"
stop-at-first-match: true
matchers:
- type: word
words:
- "PHP Extension"
- "PHP Version"
condition: andTest template for Stop-At-First-Match Attribute
Metadata attribute
A new info field has been added called metadata which can be used to provide extra metadata information in the template. This also means dynamic key: value fields now can be defined within the metadata block rather than outside of it, and the templates with additional fields in the earlier format will still work but will not be processed in the JSON output.
info:
name: Name # static field
author: test # static field
severity: info # static field
description: Description # static optional field
remediation: Remediation # static optional field
reference: https://example.com # static optional field
tags: cve,xss # static optional field
metadata:
os: linux # dynamic optional field
vendor: jira # dynamic optional fieldAdditional Field example
Scan optimization
Nuclei now track errors occurring for each Host, and if it exceeds a certain threshold defined by the max-host-error flag, the host is skipped from the scan. This saves a lot of time by not trying further template scans on unresponsive or continuously failing hosts.
Template validation enhancement
Nuclei engine now has stronger validation for loaded templates and displays the number of templates loaded with the invalid format, including errors and warnings; also, the validate flag has been upgraded to detect and report the error information to help with template fixing.
Numerous small crashes and issues have been fixed to make the overall experience smoother and better. The codebase has also been refactored to introduce new ideas to make it scalable and maintainable for the future.
GitHub issues closed in this release can be found here, and the complete changelog of the release is available on the GitHub release page.
