Communication with Black Duck services
Black Duck interacts purely as a client of the servers hosted by Black Duck at secure data centers. Black Duck understands that Black Duck is being used in conjunction with our customer's valuable software intellectual property and treats the Internet-based communications between a customer installation of our products and the data center servers with due care, ensuring that no scanned code or proprietary customer data ever leaves the customer's premises.
All information sent to the customer's Black Duck servers is treated as confidential information by Black Duck in accordance with our customer agreements. By default, all communication with Black Duck servers is done via HTTPS. More specifically, all data being transmitted is encrypted with TLS 1.3 (or TLS 1.2 if the client does not support TLS1.3). Customer sites always initiate connections with Black Duck services; the hosted Black Duck services never call out to the Black Duck application. The customer's Black Duck servers contain the HTTPS certificate, the Black Duck application initiates all connection requests using the certificate’s public key.
The information that is sent to the KB web services is for retrieving other information needed by the Black Duck application, for example, open-source metadata. Black Duck does not retain any sensitive customer application data on its servers. Customer data is used by the KB matching service and retained on the server only during this ephemeral matching process. Once completed all customer data, including filenames, is deleted. No data is retained whatsoever.
More details on the specific communications between a customer site installation of Black Duck and the Black Duck services are included below.
Registration service
The registration service provides product activation and is used to authenticate the customer’s license key and the associated entitlements. It also collects aggregate, high-level operational metrics that are used to improve services delivered to the customer.
To perform this function, information such as the registration code, and license expiration date is exchanged between the installed product(s) at the customer's site and the Black Duck registration server.
The list of metrics collected for each license key consist of: product serial number, product version, last call home data, last scan date, scan count, total number of projects, managed code base size in GB, number of code locations, total user count, hard limit of total projects authorized by the license, hard limit of total users authorized by the license, hard limit of codebase size authorized by the license, total memory being used by the Black Duck server, number of CPUs being used by the Black Duck server, and the expiration date of the license.
KnowledgeBase services
Via web services, the Black Duck KB provides the Black Duck application with the most up-to-date information about open-source software without requiring regular updates on your Black Duck server. New vulnerability data is added to the KB hourly; new searchable open-source components are added daily; and new signatures for matching are added 2-3 times per week. Other big data services are updated quarterly.
The signatures of your scanned code are used to match signatures of the open-source software in the KB to identify the open-source software contained in your code. These signatures consist of one-way, encrypted MD5 or SHA1 hashes of file system meta-data and file content. The use of MD5/SHA-1 is purely for signature hashing and is not used for any cryptographic purpose. These signatures (collections and hashes) are sent by the Black Duck application to the KB service. No source or binary code is ever sent off premises. The signatures are formatted as a JSON document and transmitted over HTTPS connections. The scan document can send file paths, file sizes, and some SHA-1 or clean SHA-1 signatures to the Black Duck server. Black Duck does post-processing on this data to create additional proprietary signatures.
Again, customer data sent to increase the accuracy of matching services is not persisted in the KB. It is used during the BOM matching process and deleted afterwards. For those reasons, no logs are produced during this process.
Additionally, the separation of customer data is enforced by the design of the architecture.
The matching service is described in more detail below.
Match Service Overview
The client POSTs a JSON document representing a tree structure of scanned nodes, with a set of signatures per node to the match endpoint. The endpoint determines if there are any matches to those signatures. If there are, it returns a JSON document with the same nodes that were sent to the match service, and adds to the document the matches from the KB for each node.
URL Format
https://<server>/kbmatch/api/v1/matches/<end-point>
Header
X-BDS-AuthToken:<>
Request Body Data
Response
As can be seen from the example above, matching open-source components are returned to the Black Duck application using GUIDs. These GUIDs are then used to retrieve metadata on each of the open-source components using KB services for vulnerabilities, licenses, and other information.
KB feedback service
In the same way that signatures and hashes are sent up to the match service, occasionally these same pieces of information will be sent associated with a specific component and version. This is done in cases where users have corrected (“adjusted”) an identification. This “vote” is recorded absent any customer context whatsoever and is used simply to increase the accuracy of the matching capability. It is in our customers' best interest to share this data as it helps improve the overall quality for the KB. However, if it is desired to not share this data, please refer to the section “Disabling the Black Duck KnowledgeBase feedback service” in the Black Duck installation guide.
Black Duck Hosted Customer Instances Monitors
In the Black Duck Black Duck Hosted environment, there are monitors and alerts enabled for each instance.
For monitoring, we are gathering the data on this metric and store it for a period of time. This data can be used for later analysis. There may be thresholds in which informational messages are dispatched, but a response is not required or triggered for these types of monitors.
Examples of scenarios we monitor are Daily Scans and Job Instances which provide useful metrics when issue analysis is being investigate by CloudOps and/or Support.
For alerting notifications, we are monitoring the metrics with thresholds. There are low and high level thresholds which may be adjusted based on reviews of Black Duck Performance Testing Lab or changes to Black Duck functionality. Low levels do not require an immediate response. High levels require an immediate response. When a certain threshold is crossed, the CloudOps team is paged via our paging system. A technical support ticket MAY be opened depending on the alert, and how many times we have notified technical support. The Technical Support team troubleshoots the issue and informs the customer of the issue while it is being investigated.
Examples of types of alerts that trigger Support tickets are: Database CPU utilization hitting the high threshold which usually indicates that the customer is performing actions which are heavily impacting the database which could cause the instance to require rebooting or resizing. Pod Readiness is monitored which indicates that a pod has been running, but that it is not ready to process requests. When it taken too long for it to come online an alert is triggered to check the instance