PicoZ

Introduction

With the Salesforce Configure Price Quote app template you can load raw input data from Salesforce CPQ for the Lead-to-Order process, extracted to either Snowflake or SQl Server using CData Sync. The raw input data is transformed through a series of transformations that take place via dbt (data build tool) and produce the required input data used in Process Mining process apps.

Lead-to-Order process

The Lead-to-Order process is an end to end process which starts with a potential customer's intent to buy a product, up to the fulfillment of this sale. See the illustration below.

Lead Qualification

Once a lead is generated, the lead will go through a qualification process. The objective of this process is to determine whether the lead is most likely to make an actual purchase. Once the lead is qualified this will turn into an actual opportunity for a sale. In Salesforce, this is tracked via the Lead object.

Opportunity Management

As a lead qualifies into an opportunity, the sales team will work on the opportunity together with the potential customer, in order to identify the need and suggest an appropriate product or service which will meet this need. In case the opportunity is won, this will generate one or more orders in order to fulfill the won opportunity. In Salesforce, this is tracked via the Opportunity object, and the Quote object.

Quote Management Once an opportunity moves forward with a customer, products and services will be offered through quotes. In Salesforce CPQ the Quote object contains information about the products and services a customer wants to buy. You can create as many quotes as you want on a given opportunity. However many quotes your opportunity contains, only one can be designated as primary.

Order Management

Within Order management, the order lifecycle is maintained, including order capture, and fulfillment. In Salesforce, the Order object is used to maintain this process. One quote may have multiple orders if the configuration is set in CPQ package.

Contract Management

Contracts are used when orders included at least one subscription product under a contract-based renewal model. Salesforce CPQ creates a contract record for them.

Optionally, this may then extend and generate an invoice and receivables, in what is called the Lead-to-Cash process.

Entity relationship diagram

The following shows all objects that were used for the process. However not all of them would be converted as entities as they were merely used for connecting the entities that actually had activities tracked in the process.

Tags

Due Dates

Salesforce Sales Cloud configuration

Source system

Salesforce Sales Cloud with Salesforce CPQ package. The CPQ package must be installed in the Salesforce Org.

Depending on which fields have field history tracked, different change events will be recorded. Field history tracking can be configured from each object within Salesforce setup. Recommended minimal setup is having the following object and fields tracked:

• Lead: Status
• Opportunity: Stagename
• Quote: Status
• Order: Status

By default, Field History is disabled per object. In order to setup Field History Tracking for an object, the following Salesforce manual contains a step by step instruction: Salesforce - Track Field History for Standard Objects. Note: if a field is activated for tracking, the values will be tracked from that moment onwards. Any changes before that point in time will not be saved. As of February 2022, Salesforce retains field history data for up to 18 months through the org and up to 24 months via the API, and Field history tracking data doesn't count towards Salesforce org's data storage limits.

In case the history needs to be stored for a longer period of time, Field Audit Trail must be activated. This allows copying of history data into a separate big data object and allows storage for up to 10 years.

Environmental settings

• To connect via CData Sync, the user establishing the connection must be API Enabled, and must have appropriate read permissions on each of the objects which are being extracted. All fields included in the input tables must be visible for the user extracting the data.
• The user must be marked as "Marketing User" in the user settings in order to extract Campaign information.
• To allow users to create multiple orders from a quote, "Allow Multiple Orders" in CPQ Order Package Settings must be selected.
• In order to calculate amounts using historical exchange rates, Effective dated currency should be enabled in the Salesforce org, and historical exchange rates should be loaded into the system.

System specific settings

The Salesforce Configure Price Quote app template works on the process starting on the marketing campaign generation, up to the generation of customer orders, if the order includes subscription products the process adds contract to track the subscriptions.

This includes processing of the following Salesforce transactional objects:

• Campaign
• CampaignMember
• Lead
• LeadHistory
• Opportunity
• OpportunityFieldHistory
• Quote (SBQQ__Quote__c)
• QuoteHistory (SBQQ_Quote_History)
• Order
• OrderItem
• OrderHistory
• Contract
• ContractHistory
• ProcessInstance
• ProcessInstanceStep

The following additional objects which are used to add additional attribute information into the model:

• Account
• RecordType
• User
• ProcessDefinition

In case they are used, the following tables add Currency conversion to the model:

• CurrencyType*
• DatedConversionRate*

*CurrencyType, DatedExchangeRate objects are by default disabled in Salesforce orgs.

Configuring CData Sync for Salesforce Sales Cloud CPQ

Below is a description on how to use CData Sync to set up a source connection and load data into a Process Mining Salesforce CPQ Lead-to-Order process app.

In general, you should follow the steps as described in Loading data using CData Sync (Snowflake) or Loading data using CData Sync (SQL Server) to set up data loading using CData Sync. Since specific settings are required when using Salesforce CPQ, pay attention to the steps described below.

Setting up the source connection

• Select Salesforce as the source system to which you want to create a connection from the list.
• Select Basic as the Auth Scheme.
• Enter the correct User, Password, Security Token, and Login URL to set up the connection to Salesforce.
• Set API Version to 52.0 or higher.

Define the following settings in the Replicate Options section in the Advanced tab in the Job Settings panel.

• Under Destination Schema, setup the target data schema in the SQL Server or Snowflake database.
• Select the Alter Schema option.
• Select the Drop Table option.

Important: Make sure you edit the Pre-job Event.

CData Sync allows the use of environment variables in order to drive specific extraction logic in each query.

In order to setup the environment variables:

<!-- Modify environment variables here. --> <!-- Variable start_extraction_date must be populated.--> <api:set attr="out.env:start_extraction_date" value="2022-01-01" /> <!-- In case a specific end date is needed, replace the value string with the required date in yyyy-MM-dd format. --> <!-- i.e api:set attr="out.env:end_extraction_date" value="2022-02-01" --> <api:set attr="out.env:end_extraction_date" value="[null | now() | todate('yyyy-MM-dd')]" /> <api:push item="out" /> 

Important: Do not modify the api:info details that are shown by default.

In order to modify the environment variables, modify the values within the Events tab. By default, end_extraction_date will default to today's date. start_extraction_date must always be populated.

Table Replication

Once the job is correctly setup, click on Add Custom Query under the Tables tab and paste the following queries (each query needs to maintain the semicolon at the end). Make sure you save all changes.

Default Extraction

The following script is prepared for scenarios where Currency Conversions and Person Accounts are not used and are disabled. In case any of these are used, please add the scripts in the Alternative Scenarios section.

REPLICATE [Account_raw] SELECT [Id], [AccountSource], [BillingCountry], [BillingState], [CreatedById], [CreatedDate], [Industry], [Name], [OwnerId], [Ownership], [ParentId], [Rating], [Type] FROM [Account]; REPLICATE [Campaign_raw] SELECT [Id], [CreatedById], [CreatedDate], [Name], [OwnerId], [ParentId], [Status], [Type] FROM [Campaign] WHERE ([CreatedDate] >= '{env:start_extraction_date}') AND ([CreatedDate] <= '{env:end_extraction_date}'); REPLICATE [CampaignMember_raw] SELECT [Id], [CampaignId], [CreatedById], [CreatedDate], [LeadId], [Status] FROM [CampaignMember] WHERE ([CreatedDate] >= '{env:start_extraction_date}') AND ([CreatedDate] <= '{env:end_extraction_date}'); REPLICATE [Lead_raw] SELECT [Id], [ConvertedOpportunityId], [CreatedById], [CreatedDate], [Name], [OwnerId] FROM [Lead] WHERE ([CreatedDate] >= '{env:start_extraction_date}') AND ([CreatedDate] <= '{env:end_extraction_date}'); REPLICATE [Opportunity_raw] SELECT [Id], [AccountId], [Amount], [CloseDate], [CreatedById], [CreatedDate], [ForecastCategory], [IsClosed], [IsWon], [LastStageChangeDate], [LeadSource], [Name], [OwnerId], [Probability], [StageName], [Type], [SBQQ__AmendedContract__c] FROM [Opportunity] WHERE ([CreatedDate] >= '{env:start_extraction_date}') AND ([CreatedDate] <= '{env:end_extraction_date}'); REPLICATE [RecordType_raw] SELECT [Id], [Name] FROM [RecordType]; REPLICATE [User_raw] SELECT [Id], [Department], [Name], [UserType] FROM [User]; REPLICATE [ProcessDefinition_raw] SELECT [Id], [Name], [TableEnumOrId], [Type] FROM [ProcessDefinition] WHERE [Type] = 'Approval'; REPLICATE [ProcessInstance_raw] SELECT [Id], [TargetObjectId], [CreatedById], [CreatedDate], [ProcessDefinitionId], [Status] FROM [ProcessInstance] WHERE ([CreatedDate] >= '{env:start_extraction_date}') AND ([CreatedDate] <= '{env:end_extraction_date}'); REPLICATE [ProcessInstanceStep_raw] SELECT [Id], [Comments], [CreatedById], [CreatedDate], [ProcessInstanceId], [StepStatus] FROM [ProcessInstanceStep] WHERE ([CreatedDate] >= '{env:start_extraction_date}') AND ([CreatedDate] <= '{env:end_extraction_date}'); REPLICATE [SBQQ__Quote__c_raw] SELECT [Id], [CreatedById], [CreatedDate], [Name], [SBQQ__Opportunity2__c] FROM [SBQQ__Quote__c] WHERE ([CreatedDate] >= '{env:start_extraction_date}') AND ([CreatedDate] <= '{env:end_extraction_date}'); REPLICATE [Order_raw] SELECT [Id], [CreatedById], [CreatedDate], [Name], [OpportunityId], [SBQQ__Quote__c] FROM [Order] WHERE ([CreatedDate] >= '{env:start_extraction_date}') AND ([CreatedDate] <= '{env:end_extraction_date}'); REPLICATE [OrderItem_raw] SELECT [OrderId], [SBQQ__Contract__c] FROM [OrderItem] WHERE ([CreatedDate] >= '{env:start_extraction_date}') AND ([CreatedDate] <= '{env:end_extraction_date}'); REPLICATE [Contract_raw] SELECT [Id], [CreatedDate], [ContractNumber], [CreatedById], [OwnerId] FROM [Contract]; REPLICATE [LeadHistory_raw] WITH ReplicateStartDate = '{env:start_extraction_date}', replicateenddate = '{env:end_extraction_date}' SELECT * FROM [LeadHistory] WHERE ((((([Field] = 'Status') OR ([Field] = 'Rating')) OR ([Field] = 'Owner')) AND ([DataType] != 'EntityId'))); REPLICATE [OpportunityFieldHistory_raw] WITH ReplicateStartDate = '{env:start_extraction_date}', replicateenddate = '{env:end_extraction_date}' SELECT * FROM [OpportunityFieldHistory] WHERE (((((([Field] = 'Amount') OR ([Field] = 'StageName')) OR ([Field] = 'Owner')) OR ([Field] = 'CloseDate')) AND ([DataType] != 'EntityId'))); REPLICATE [SBQQ__Quote__History_raw] WITH ReplicateStartDate = '{env:start_extraction_date}', replicateenddate = '{env:end_extraction_date}' SELECT * FROM [SBQQ__Quote__History] WHERE ((((([Field] = 'SBQQ__Status__c') OR ([Field] = 'SBQQ__ExpirationDate__c')) OR ([Field] = 'SBQQ__Primary__c')) OR ([Field] = 'SBQQ__SubscriptionTerm__c')) OR ([Field] = 'SBQQ__SalesRep__c')) AND ([DataType] != 'EntityId'); REPLICATE [OrderHistory_raw] WITH ReplicateStartDate = '{env:start_extraction_date}', ReplicateEndDate = '{env:end_extraction_date}' SELECT * FROM [OrderHistory] WHERE ((([Field] = 'Status') OR ([Field] = 'Owner')) OR ([Field] = 'TotalAmount')) AND ([DataType] != 'EntityId'); REPLICATE [ContractHistory_raw] WITH ReplicateStartDate = '{env:start_extraction_date}', replicateenddate = '{env:end_extraction_date}' SELECT * FROM [ContractHistory] WHERE ((([Field] = 'Status') OR ([Field] = 'Owner')) OR ([Field] = 'ContractTerm')) AND ([DataType] != 'EntityId'); 

Note regarding CData Salesforce Provider Version 21.0.8097.0 When a table has an _Incremental Check Column_ field predefined in CData Sync, the SQL query cannot use the same date field in both statements (WITH and WHERE) at the same time. If version 21.0.8097.0 (or an older version) of CData is used, the date filtering should be done in the WHERE clause and should be removed from the WITH statement.

Alternative Scenarios

The following scenarios can be used in combination of each other:

Multi Currency and Dated Conversion Rates are enabled in Salesforce Org

In case multi currency is used in the Salesforce org, add the following queries:

REPLICATE [CurrencyType_raw] SELECT [Id], [IsCorporate], [IsoCode] FROM [CurrencyType] WHERE [IsCorporate] = 1; REPLICATE [DatedConversionRate_raw] SELECT [Id], [IsoCode], [NextStartDate], [StartDate], [ConversionRate] FROM [DatedConversionRate]; 

Additionally, the Opportunity query must be changed to the following to add the CurrencyIsoCode field in the query.

REPLICATE [Opportunity_raw] SELECT [Id], [AccountId], [Amount], [CloseDate], [CreatedById], [CreatedDate], [CurrencyIsoCode], [ForecastCategory], [IsClosed], [IsWon], [LastStageChangeDate], [LeadSource], [Name], [OwnerId], [Probability], [StageName], [Type], [SBQQ__AmendedContract__c] FROM [Opportunity] WHERE ([CreatedDate] >= '{env:start_extraction_date}') AND ([CreatedDate] <= '{env:end_extraction_date}'); 

Person Accounts enabled in Salesforce Org

In Case Person Accounts (field IsPersonAccount) are used - replace the Accounts query with the following query. This query will add the field IsPersonAccount in the Accounts extraction.

REPLICATE [Account_raw] SELECT [Id], [AccountSource], [BillingCountry], [BillingState], [CreatedById], [CreatedDate], [Industry], [IsPersonAccount], [Name], [OwnerId], [Ownership], [ParentId], [Rating], [Type] FROM [Account]; 

Record Types are enabled in Salesforce Org

In case Record Types are used the Opportunity query must be amended to add the RecordTypeId field in the query. The following replicate shows Opportunity with RecordTypeId and CurrencyIsoCode values added.

REPLICATE [Opportunity_raw] SELECT [Id], [AccountId], [Amount], [CloseDate], [CreatedById], [CreatedDate], [CurrencyIsoCode], [ForecastCategory], [IsClosed], [IsWon], [LastStageChangeDate], [LeadSource], [Name], [OwnerId], [Probability], [RecordTypeId], [StageName], [Type], [SBQQ__AmendedContract__c] FROM [Opportunity] WHERE ([CreatedDate] >= '{env:start_extraction_date}') AND ([CreatedDate] <= '{env:end_extraction_date}'); 

Input fields

The following tables include the list of fields per input table, their description, data type to be used when formatting the input and the filter flag to identify those that are being used to filter data.

Input types

Below is an overview of the different field types and their default format settings.

Account

Represents an individual account, which is an organization or person involved with your business (such as customers, competitors, and partners).

Campaign

Represents and tracks a marketing campaign, such as a direct mail promotion, webinar, or trade show.

CampaignMember

Represents the association between a campaign and either a lead or a contact.

Contract

Represents the subscription products that customers have purchased. The Contract store information on the subscriptions that sales reps have quoted or ordered.

ContractHistory

History for tracked fields of Contract.

CurrencyType

Represents the currencies used by an organization for which the Multi-currency feature is enabled.

DatedConversionRate

Represents the dated exchange rates used by an organization for which the Multi-currency and the effective dated currency features are enabled.

Lead

Represents a prospect or lead.

LeadHistory

History for tracked fields of Lead.

Opportunity (Main Object)

Represents an opportunity, which is a sale or pending deal. This will be the main Case Identifier for the Process Mining model.

OpportunityFieldHistory

Represents the history of changes to the values in the fields of an opportunity.

Order

Represents an order associated with a quote, an order record track the products from customers purchases.

OrderItem

Represents an order product which is a product or service that is provided to a customer according to an associated order for each of the quote lines.

Note: Filter in OrderItem is done by CreatedDate field which is not being extracted

OrderHistory

Represents historical information about changes that have been made to the standard fields of the associated order, or to any custom fields with history tracking enabled.

Quote

Represents the Quote object that stores all the details of the products and prices quoted for an opportunity. Quotes are created from and synced with opportunities, and emailed as PDFs to customers.

QuoteHistory

Represents historical information about changes that have been made to the standard fields of the associated quote, or to any custom fields with history tracking enabled.

ProcessDefinition

Represents the definition of a single approval process.

ProcessInstance

Represents an instance of a single, end-to-end approval process.

ProcessInstanceStep

Represents one work item in an approval process (ProcessInstance).

RecordType

Represents a record type, which might define a different sales process for opportunities.

User

Represents each user in the Salesforce organization.

Design specifications

Entities

Salesforce already treats each object as an individual entity, however the objects are joined to additional master data in order to provide additional information.

Activities

*The actual activity names will depend on the values defined on the customer Salesforce organization.

Fields

Cases_base

*Case attributes which need to be configured on top of standard TemplateOne case attributes.

Event_log_base

*Event attributes which need to be configured on top of standard TemplateOne event attributes.

Customizing the transformations

Configuration

Automated Users

In order to add data which will be defined by the end user, one additional file is loaded into the model which need to be prepared by the Analyst. This is the Automated_users_raw file, which contains the users that are flagged as automated. If this is not configured, the connector will still work properly but it will not have that extra information.

Amount of events

All contracts that get amended will generate automatically a new Opportunity_id that will be linked to the former. Therefore, all activities happening to the contract will be included in both case_id's: the original opportunity and the amended one. That can be seen when comparing the total amount of records in Events_all table vs Event_log_base table, thus it will be correct to have more records in the Event_log_base table.

Dbt Variables

Below is an overview of the variables that must be configured based on the Salesforce configuration for the end customer. These variables are defined in the dbt_project.yml file.

Limitations

CDATA Sync - Salesforce Connector Regarding CData Salesforce Provider Version 21.0.8097.0 When a table has an Incremental Check Column field predefined in CData, the SQL query cannot have both statements (WITH and WHERE) at the same time. If this is the version of CData that is being used, please remove the entire WHERE statement so the date filter done in the WITH statement works properly.

No Event Start

Salesforce does not store the start time for each change activity.

No flag to identify automated users

Salesforce does not have a field or special license for automated users. It does however have a default user named Automated Process which is used for automated processing for Salesforce changes, however any additional users such as interfaces or RPA will use normal licenses and are not differentiated by user type.

Salesforce Retention Policy

As of February 2022, Salesforce retains field history data for up to 18 months through the org and up to 24 months via the API.

In case the history needs to be stored for a longer period of time, Field Audit Trail must be activated. This allows copying of history data into a separate big data object and allows storage for up to 10 years.

Troubleshooting

General

Field not available in object during data extraction

Check Field Level Security in order to confirm the field is available and visible for the user profile extracting data. For CurrencyISOCode, IsPersonAccount, LastStageChangeDate, RecordTypeId check the specific issues in the following section.

Object not available during data extraction

The user profile might not have read access to the object. This can be corrected on the user profile or a different user should extract the data. For Order, Quote, CurrencyType, DatedConversionRate check the specific issues in the following section.

Object Specific

Campaign and CampaignMember objects not available

Check the user record for the user extracting data, and verify the Marketing User checkbox is Active.

CurrencyIsoCode is not available when extracting data from the Opportunity object

Multi-currency is not enabled within the Salesforce org. In case this happens, the conversions should be removed from the data model. This can be achieved by adjusting the variable use_currency_conversion in the dbt_project.yml to false.

CurrencyType object is not available when extracting data

Multi-currency is not enabled within the Salesforce org. In case this happens, the conversions should be removed from the data model. This can be achieved by adjusting the variable use_currency_conversion in the dbt_project.yml to false.

DatedConversionRate object is not available when extracting data

Effective currency rate management is not enabled within the Salesforce org. In case this happens, the conversions should be removed from the data model. This can be achieved by adjusting the variable use_currency_conversion in the dbt_project.yml to false.

IsPersonAccount field not available in Account object when extracting from Salesforce

When extracting data from Salesforce, IsPersonAccount may not be available as the organization may not have B2C customers. In case this happens, the field should be removed from the data model. This can be achieved by adjusting the variable use_is_person_account in the dbt_project.yml to false.

LastStageChangeDate field not available in Opportunity object when extracting from Salesforce

The field LastStageChangeDate was only made available starting with Salesforce REST API 52.0. Make sure the CData extraction or the extraction method used is using API 52.0 onwards.

RecordTypeId is not available when extracting data from the Opportunity object

Record Types are not used within the Salesforce org. In case this happens, the conversions should be removed from the data model. This can be achieved by adjusting the variable use_record_types in the dbt_project.yml to false.

ncG1vNJzZmicn5jCrrHNrZitoZ%2Bjvaqv066pnqqVpbxvrsuomWebn6eyb8PIp5uor6Nju6bAjp%2BgpZ2jZJ2zu8KeqqyXfZ67qrrGaGlpamJjfnF7spqjnquWpL%2BksYx8pqeemZzCs7GMiamim5WUnra7055mjJmcmsCnu9GcnGZ7n6OzqrPUq5xmiKKesKZ5sK6mrZ1encGuuA%3D%3D