Running a component scan using the Signature Scanner command line

You run a component scan to identify the components contained in an archive or a directory of files.

Note: If the client running the component scans needs to communicate to Black Duck via a proxy server, you must set a SCAN_CLI_OPTS environment variable prior to running the client.
Note: An error message appears if you exceed the scan size limit, which is 5 GB. Contact Customer Support if you receive this message.

The usage is:

scan.cli.bat [parameter1]...[parameterN]...<scan_path>
Parameter Description
-?, --help Shows help for this tool.
<scan_path> Path to the file directory location or archive that you want to scan.

--bdio2

Use BDIO 2's JSON-LD format for storing and transmitting scan evidence to the Black Duck server.

--binaryAllowedList <file extensions>

--sourceAllowedList <file extensions>

Use to create approved signature lists.

  • --binaryAllowedList x, y, x where x, y, z are the approved file extensions for SHA-1 (binary) files.

  • --sourceAllowedList a, b, c where a, b, c, are the approved file extensions for clean SHA-1 (source code) files.

--cloneFrom <version>

Specifies the name of an existing project version to use as a clone.

To clone a project version, use the:

  • --project parameter to specify the project you wish to clone from.

  • --release parameter to specify the new project version.

  • --cloneFrom parameter to specify the project version to use as a clone.

For example, to clone version 1.0 of project SampleProject to a new version called 2.0, you would include these parameters:

--project SampleProject --release 2.0 --cloneFrom 1.0
--context <context> Additional URL context. Use this parameter, for example, if the X-Forwarded-Prefix header is being specified in a proxy server/load balancer configuration.
--copyright-search Enables copyright text detection.
--correlationId <value> Used by the Black Duck system to provide the ability to correlate results from different types of scan (package manager scan and signature scan) for the same code location to improve the reliability of results.
--dryRunReadFile <data directory> Specifies the directory, including the file name, from a dryRun scan and posts the scan to the Black Duck server.
--dryRunWriteDir <data directory> Specifies the directory to which the scanner outputs a BDIO file with the original file metadata used for scanning. The scanner does not connect to or post the scan to the Black Duck server. Note that the data directory is created inside the specified directory.
--exclude <pattern>

--exclude-from <Filename>

Excludes a directory or several directories from scanning.

The scanner automatically excludes these directories and the contents of these directories:

  • CVS
  • .svn
  • .git
  • .hg
  • .bzr
  • __MACOSX

The scanner automatically excludes files named:

  • .cvsignore
  • .git
  • .gitignore
  • .gitattributes
  • .gitmodules
  • .hgignore
  • .hgsub
  • .hgsubstate
  • .hgtags
  • .bzrignore
  • vssver.scc
  • .DS_Store

To exclude other directories, use --exclude to exclude a single directory; --exclude-from to specify a file that lists directories that should be excluded.

Exclusion guidelines:

  • Leading and trailing forward slashes are required.

    For example, if you enter exclude /directory, a warning message will appear and the directory will not be excluded. If you enter /directory in the file, the directory will not be excluded.

  • Directory names cannot contain double asterisks (**).

  • Specify one directory per line in the file. Include the complete direct path. The path must be a relative path rather than an absolute path.

  • You cannot exclude archives or contents within archives.

There are two additional methods you can use to exclude directories from scanning:

  • Create an ignore file located in the $HOME/config/blackduck directory.

    Use this file to list excluded directories, relative to root. This option provides you with the ability to use one location to list all directories that need to be excluded.

    Lines in the ignore file must have the path from source root to the ignored directory, may have multiple subdirectories, and must have leading and trailing forward slashes (/).

    Create one of the following environment variables, as shown here, configured for Linux or Mac OS X:

    • export JAVA_TOOL_OPTIONS=" -Duser.home=<path>"

      The .ignore file must be located here: <user.home>/.config/blackduck/ignore

    • export XDG_CONFIG_HOME=<path>

      The .ignore file must be located here: $XDG_CONFIG_HOME/blackduck/ignore

    • export JAVA_TOOL_OPTIONS="-Dblackduck.scan.excludesFile=<path>/<file>"

      The .ignore file can be in any location and have any name.

    For Windows systems, use the Control Panel to access the Advanced System Settings dialog box to create the environment variables.

  • Create individual .bdignore files which can be located in any directory.

    Use this file to list the excluded subdirectories in the directory where the .bdignore file is located. You must create a .bdignore file in each directory that has subdirectories you want to exclude.

You must also follow the exclusion guidelines as described above when using either of these methods.

Tip: Use the debug parameter when excluding directories to ensure that the scanner visited and excluded the directory.
--fs-wait-time <number of minutes>

Number of minutes the scan client waits for the File System scan (signature scan) to be in either the completed or error status before starting the snippet or string search scan. Must be a positive integer number.

--host <host> Server hosting the Black Duck installation.
--individualFileMatching <option>

Individual file matching is the identification of a component based purely upon the checksum information of a single file. By default, individual file matching is disabled. To enable individual file matching, select one of the following options:

  • source. Performs individual file matching only on files with this extension: .js, c, h, c+, cc, cpp, cxx, hh, hpp, hxx, h+, cs, idl, rc

  • binary. Performs individual file matching on files with these extensions: .apklib, .bin, .dll, .exe, .o, and .so.

  • all. Performs individual file matching on all files with extensions matching "*".

  • both. Performs individual file matching on file extensions only using both approved signature lists.

Any other value will be ignored and individual file matching will remain disabled.

Click here for more information on using this parameter with approved signature lists.

--insecure Ignores TLS validation errors, allowing the scanner to connect to the Black Duck server.
--license-search Enables searching for embedded licenses.
--logDir <log directory>

Location of the log directory which contains all scanner log files. You must specify the --logDir parameter for log files to be saved.

--matchConfidenceThreshold <value> Specify the match confidence threshold as a percentage value between 1 - 100. A low value returns more matches; whereas a high value returns fewer matches.

--max-request-body-size <size>

--max-update-size <size>

Controls how scan data is streamed (buffered) from the Signature Scanner to Black Duck.

In rare cases, you may need to modify these values to better suit your network, for example, decreasing the values if there are issues with your network or increasing the default values if your network is highly stable.

  • --max-request-body-size. Size of the main request that uploads the scan data for scanned paths.

    Specify a value, in bytes. The default is 20000000 bytes. The recommended minimum value is 2000000 bytes; the recommended maximum value is 2000000000 bytes.

  • --max-update-size Buffers an update request to inform Black Duck when Signature Scanner has completed uploading the data of individual URIs (scanned paths).

    Specify a value, in bytes. The default value of 10000 bytes. The recommended minimum value is 1000 bytes; the recommended maximum value is 1000000 bytes.

--min-scan-interval=<time in hours>

Minimum scan interval setting (in hours), which may be used to limit daily rate of the signature scan for given code location. If set to greater than 0, signature scans will not be processed if they occur before the set scan interval.

--name <scan name>

Unique name identifying this scan. This name is displayed on the Scans page. Click here for more information.

Note: The --name parameter is not supported when specifying multiple scan paths in a single command line.
--no-prompt

Non-interactive mode.

Instead of the --no-prompt parameter, you can set the BD_HUB_NO_PROMPT environment variable to enable non-interactive mode.

--no-signature-generation Use the traditional Signature Scanner instead of the Enhanced Signature Generation.
--password <password>

Forces the scanner to prompt you for the password for the user account with the code scanner role:

  • Specifying the --password parameter without the password value results in the scanner prompting you for the password.

  • Specifying the password value displays a warning message notifying you that specifying the password on the command line will not be supported in future versions of Black Duck; the scan then runs.

Set the BD_HUB_PASSWORD environment variable with the Black Duck server password instead of passing an argument to the --password parameter:

  • If you set this environment variable and specify the --password parameter, the scanner prompts you for the password; it does not check the password value against the value specified in the environment variable.

  • If you set this environment variable and do not specify the --password parameter, the scanner does not prompt you for the password.

Important: Set the BD_HUB_PASSWORD environment variable with the Black Duck server password. If you supply the password parameter, an error message appears and the scan will not complete.

If this environment variable is not set, the scanner prompts you for the password whether you include or omit the --password parameter.

Note: If the password parameter is the parameter immediately before <scan_path> use -- to indicate you are finished passing parameters, for example --password -- <scan_path>. Otherwise, the scanner will try to use the <scan_path> value as the password.

Instead of specifying a username and password, use the BD_HUB_TOKEN environment variable to specify a Black Duck API token.

--port <port>

Port on which the Black Duck server instance is listening.

--project <project>

Name of the project to which you want to map the scan results.

If you specify a project, you must specify a version.

  • If the project and project version exist, the scanner maps or remaps the scan results.

  • If the project exists, but the version does not, the scanner creates the version and maps the scan results.

--project-group <project group name>

Assigns the project to the designated project group. If the project does not already exist, it is created in the corresponding project group.

This parameter has no effect if the project already exists or if the specified project group does not exist.

--release<release>

Name of the project version to which you want to map the scan results.

If you specify a version, you must specify a project.

  • If the project and project version exist, the scanner maps or remaps the scan results.

  • If the project exists, but the version does not, the scanner creates the version and maps the scan results.

--retain-unmatched-files

--discard-unmatched-files

Used to retain or discard, respectively, any unmatched files discovered by this scan and this scan only. If either option is supplied, project and global retention settings are ignored; otherwise, retention is determined by project or global settings as described in "Settings". Specifying both options with a single scan is an error.
--scheme <scheme>

Protocol to use to connect to the server hosting the Black Duck installation. Possible values are http or https; https is the default value. You must include -scheme https to specify the https protocol.

--statusWriteDir <directory> Specifies the directory to which the scanner outputs a JSON file which contains the complete scan status information.
--selfTest Performs a self-test; will not connect to or post the scan to the Black Duck server.

--snippet-matching

--snippet-matching-only

--snippet-matching-all-source

--full-snippet-scan

Select one of the following for snippet matching:

  • --snippet-matching. Selecting this parameter enables a two-phase approach to scanning. First, a component scan is completed whereby only files that have changed since the previous scan are scanned. Once that component scan is completed, a snippet scan runs on those newly scanned files only: if a previously scanned file has not changed, it will not be rescanned for snippets.

    Black Duck Software recommends using this parameter for snippet scanning.

  • --snippet-matching-only. Selecting this parameter runs a snippet scan only on files that have changed; a component scan is not performed.

    You must have successfully completed a full file scan prior to selecting this parameter.

  • --snippet-matching-all-source. Selecting this parameter runs a snippet scan for all files with supported extensions (whether they belong to unmatched directories/archives or not).

  • --full-snippet-scan. Selecting this parameter forces the snippet scan to search the KnowledgeBase regardless of locally cached matches from previous snippet scans.

    This parameter must be used with the --snippet-matching, --snippet-matching-only, or the --snippet-matching-all-source parameter:

    • With the --snippet-matching parameter: First, a signature scan is completed. Once that scan is completed, a snippet scan is performed on unmatched files or files belonging to unmatched directories/archives

    • With the --snippet-matching-only parameter: A snippet scan is performed on unmatched files or files belonging to unmatched directories/archives; a signature scan is not executed, but it must already exist.

    • With the --snippet-matching-all-source. First, a signature scan is completed. Once that scan is completed, a snippet scan is performed for all files with supported extensions (whether they belong to unmatched directories/archives or not).

To upload source files, you must use the --upload-source parameter, as described below.

--tlscertpass

Forces the scanner to prompt you for the password for the client certificate.

You can specify the --tlscertpass parameter and/or set the BD_HUB_CLIENTCERT_PASS environment variable which specifies the private key password for the client certificate, for example, when --tlscert points to an encrypted PKCS #12 key store.

The result of specifying the --tlscertpass parameter depends on whether the key is encrypted.

  • If the key is encrypted, the scan will fail if you do not set the BD_HUB_CLIENTCERT_PASS environment variable or specify the --tlscertpass parameter.
    • If you set the environment variable and specify the --tlscertpass parameter, the scanner prompts you for the password; it does not check the password value against the value specified in the environment variable.

  • If the key is not encrypted, regardless of whether the BD_HUB_CLIENTCERT_PASS environment variable is set:
    • Specifying the --tlscertpass parameter forces the scanner to prompt you for the password for the client certificate. The scan will fail unless the password is empty.

    • If you do not specify the --tlscertpass parameter, the scan will succeed.

--tlskey <keyFile>

Black Duck client certificate private key file. Automatically sets --scheme to https.

Note: This parameter is optional as the key and certificate can be in included in the key store file specified with --tlscert.
--tlscert <certFile>

Black Duck client certificate chain file or key store file. Automatically sets --scheme to https. Click here for more information on using certificate-based authentication.

--upload-source

Uploads the source file, an optional feature for snippet matching, embedded license search, and copyright text search.

This parameter is optional with the --snippet-matching or --snippet-matching-only parameters or with the --license-search and/or --copyright-search parameters.

--username <username>

Black Duck user account with the code scanner role.

Instead of specifying a username and password, use the BD_HUB_TOKEN environment variable to specify a Black Duck API token.

-V, --version Shows the version information of this tool.
-v, --verbose Sets the logging level to verbose.
--debug Shows debug output.

Other environment variable:

  • BD_HUB_TOKEN

Used to specify the Black Duck API token which is the preferred authentication method over username and password.

Use the Profile page in the Black Duck UI or the api-token-rest-server API to manage API tokens.

Specifying the password

Set the BD_HUB_PASSWORD environment variable with the Black Duck server password. If you supply the password parameter, the scan will not complete.

About package management files

By default, the scanner does not include components declared in supported package management files. Use Black Duck Detect to discover declared dependencies.

Exit Statuses

The possible exit statuses are:

  • 0: SUCCESS. The export completed successfully.

  • 1: FAILURE. Generic failure.

  • 2: NOT_EXECUTED. Returned by the scan client when the configured minimum scan interval is not exceeded and the scan was not executed. See the --min-scan-interval command line argument above for additional details.
  • 64: USAGE. The command to run the tool was used incorrectly, for example, with the wrong number of arguments or a bad syntax.

  • 65: DATA_ERROR. The input data was incorrect.

  • 66: NO INPUT. An input file (not a system file) did not exist or was not readable.

  • 67: NO_USER. The specified user does not exist.

  • 68: NO_HOST. The specified host does not exist.

  • 69: UNAVAILABLE. A service is unavailable.

  • 70: SOFTWARE. An internal software error has been detected.

  • 71: OS_ERROR. An operating system error has been detected.

  • 72: OS_FILE. A system file does not exist, cannot be opened, or has some sort of error, for example a syntax error.

  • 73: CANNOT_CREATE. An output file cannot be created.

  • 74: IO_ERROR. An error occurred while doing input/output on a file.

  • 75: TEMPORARY FAILURE. Temporary failure,

  • 76: PROTOCOL. The remote system returned something that was "not possible" during a protocol exchange.

  • 77: NO_PERMISSION. You did not have sufficient permission to perform the operation.

  • 78: CONFIGURATION. Something was found in an unconfigured or misconfigured state.

  • 79: NO_REGISTRATION. Registration to Black Duck or Protex was not valid.

You can also find more information about these exit codes here.

Examples

The following are examples of using the command line to run the Signature Scanner CLI.

  • Scanning and sending scan data toBlack Duck

  • Scanning and mapping the scan data

Note that:

  • In all examples, the user has a code scanner role. Contact your Black Duck administrator for more information.

  • The examples show only required parameters.

To scan and send the scan data to Black Duck:

  1. Open a command prompt.

  2. Go to the directory where the Signature Scanner is installed.

    For example:

    Linux/MAC OSX:

    /opt/blackduck/hub/scan.cli-2024.7.3/scan.cli-2024.7.3/bin

    Windows:

    C:\scan.cli-2024.7.3\scan.cli-2024.7.3\bin

  3. Run the following command to configure and initiate the scan.

    For example:

    Linux/Mac OSX:

    ./scan.cli.sh --username <username> --host <host> --port <port> <scan_path>

    Windows:

    scan.cli.bat --username <username> --host <host> --port <port> <scan_path>

    The Signature Scanner sends the scan data to Black Duck's server. Log in to Black Duck to map the component scan to a project, which adds the identified components to the project BOM.

To scan over HTTPS, sending the scan data to Black Duck, and automatically mapping scan to a project:

  1. Open a command prompt.

  2. Go to the directory to which the scanner is installed.

    Linux/MAC OSX:

    /opt/blackduck/hub/scan.cli-2024.7.3/scan.cli-2024.7.3/bin

    Windows:

    C:\scan.cli-2024.7.3\scan.cli-2024.7.3\bin

  3. Run the following command to configure and initiate the scan.

    Linux/Mac OSX:

    ./scan.cli.sh --username <username> --host <host> --port <port> --scheme HTTPS --project <project> --release <release> <scan_path>

    Windows:

    scan.cli.bat --username <username> --host <host> --port <port> --scheme HTTPS --project <project> --release <release> <scan_path>

The Signature Scanner sends the scan data to the Black Duck server and automatically maps the scan to the version of the project you specified.