Tuesday, February 1, 2011

Variables Window

Use the Variables window to create and modify user-defined variables and view system variables.
You can optionally display the Variables window by mapping the View.Variables command to a key combination of your choosing on the Keyboard page of the Options dialog box. 


Add Variable

Add a user-defined variable.
Delete Variable
Select a variable from the list, and then click Delete Variable.
Show System Variables
Click to toggle between a list that includes system variables and a list that does not. System variables are predefined. You cannot add or delete system variables or modify their properties. This list is color coded. System variables are gray, and user-defined variables are blue. The list includes variable properties.
Show All Variables
Click to toggle between a list that includes all the variables in a package and a list that filters the variables. If the Show All Variables option is not selected, the list includes only the variables in the scope of the package or the selected container or task; otherwise, the list includes all variables. If the Show System Variables option is selected, the list includes system variables. System variables are predefined. You cannot add or delete system variables or modify their properties. This list is color coded. User-defined variables are blue and system variables are gray. The list includes variable properties.
Choose Variable Columns
Click to open the Choose Variable Columns dialog box where you can change the column selection. Use the Variable Column Chooser dialog box to select the columns whose properties will display in the Variables window.
 
Scope
Select to display the scope of variables.
Data type
Select to display the data types of variables.
Value
Select to display the values of variables.
Namespace
Select to display the namespaces of variables.
Raise event when variable value changes
Select to display Booleans that indicate whether the variables raise the OnVariableValueChanged event.

Name


View the variable name. For user-defined variables, you can update the variable name.

Scope


View the scope of the variable. A variable has either the scope of the entire package, or the scope of a container or task. The scope of the variable must be sufficient so that the variable is visible to any other tasks or components that need to read or set its value. Scope cannot be modified in the Variables window, but depends on the item that was selected in the designer when you created the variable.

Data Type


View the data type of the variable. For user-defined variables, you can select a data type from the list.

Value


View the variable value. For user-defined variables, you can update the variable value. This value can be a literal or an expression.

Namespace


View the namespace name. User-defined variables are initially created in the User namespace, but you can change the namespace name. To display this column and change the namespace, click Choose Variable Columns and select the option in the Choose Variable Columns dialog box.

Raise Change Event


Indicate whether to raise an event when a value changes. By default, the Variables window does not list this column. Click Choose Variable Columns and select the option in the Choose Variable Columns dialog box.

IncludeInDebugDump


Indicate whether the variable value is included in the debug dump files.
For user-defined variables and system variables, the default value for the InclueInDebugDump option is true.
However, for user-defined variables, the system resets the IncludeInDebugDump option to false when the following conditions are met:
  • If the EvaluateAsExpression variable property is set to true, the system resets the IncludeInDebugDump option to false.
    To include the text of the expression as the variable value in the debug dump files, set the IncludeInDebugDump option to true.
  • If the variable data type is changed to a string, the system resets the IncludeInDebugDump option to false.
When the system resets the IncludeInDebugDump option to false, this might override the value selected by the user.

Using SQL Server Configurations to configure Flat File Connection Manager



The following is an example on how to use SQL Server based configurations to configure a Flat File Connection Manager’s ConnectionString property. We will store in an SSIS variable the directory that the Flat File Connection Manager will store/read its file to/from. The file name in this example will be based on another variable. We will use SQL Server configurations to dynamically manage the directory in this case, though you could/should do both.
  • Create a new package.
  • Create a new string variable, “Directory” and give it a default value of “C:\”
  •  Create another new string variable, “FileName” and give it a default value of “test.txt”
NOTE: You don’t need the quotes when typing in the values for the above variables
  •    Right click on the Control Flow, select “Package Configurations…”
  • Check the box to “Enable package configurations”
  • Click “Add…”
  • Change “Configuration type:” to “SQL Server”
  • Click on “New…” on the right of the “Connection:” line to create a new connection.
  • This will bring up a new window, “Configure OLE DB Connection Manager”. Click “New…” here.
  • Enter in the required information on this screen such as server (“localhost” in my case) and database name (“AdventureWorksDW” in my case). I am using “Windows Authentication”
  • Click “OK”
  • Click “OK” again
  • Now we should be back to the “Package Configuration Wizard” screen. Click “New…” on the right of the “Configuration table:” line to create a new configuration table (if you don’t already have one created.)
  • This brings up a “Create Table” screen. Leave the SQL alone, unless you want to change the name of the table it is trying to create. (Which I do — I replace the space with an underscore: “SSIS_Configurations”) Click “OK”
  • Back on the “Package Configuration Wizard” screen, you can type in a “Configuration filter:” value. This is basically a WHERE clause when SSIS tries to query the table from above. On the next screen, we’ll get to choose which properties we’d like to configure via this method, so give this a unique name for this test package. I used “test_package”.
  •   Click “Next >”
  • Collapse all branches on this next window in the “Objects” pane. Make sure that nothing is checked. This is easy to do by checking the top level object (the name of your package) and unchecking it again. Expand the “Variables” sub-tree. Here you’ll see the two variables we created from before. Expand the “Directory” variable. Expand its “Properties” branch. Check the “Value” property under the “Directory” branch. At this point, you should only have one item checked.
  • Click “Next >”
  • Give this configuration a name. This is only used to help you identify which configuration is which in this package, as you can have more than one at any given time. I used “Directory Configuration” in this case. Click on “Finish”
  • Click on “Close”
  •           From here, we have the configuration setup, which will dynamically set the value of the “Directory” variable based on whatever value is in the configuration table. Now let’s set it up on a Flat File Connection Manager object.
·          
  •  In the “Connection Managers” pane at the bottom of the window, right click and select “New Flat File Connection…”
  •  Give it a “Connection manager name” of something descriptive. I used “My Test Flat File” and left the “Description” property blank.
  • Type in the name of the file, fully qualified. I used “C:\test_placeholder.txt”
  •  I used the default options of “Format: Delimited” and “Text qualifier: ” and “Header row delimiter: {CR}{LF}” and “Headers rows to skip: 0″
  • Click on the “Advanced” tab on the left.
  • Click on the “New” button at the bottom to add a new column. Click on it again to add a second column.
  • Click “OK”
Now the Flat File Connection Manager has been setup, though it is not configured to use the variables from before. Let’s do that now.
  •  Right click on the Flat File Connection Manager object in the “Connection Managers” pane and select “Properties”
  •    Click on the “Expressions” property. If you have installed SQL Server 2005 Service Pack 1 or greater, you should now see some ellipses (…) on the right. Click on them.
  • You should now have the “Property Expression Editor” showing. In the left side, click and select the “ConnectionString” property.
  •   Click on the ellipses here as well to bring up the expression editor for this property.
  • Use the following for your expression: @[User::Directory] + @[User::FileName]
  • Click “OK”
  • Close the properties window if you want.
  • Now we have the Flat File Connection Manager object configured to use our variables, which will get their values from our database table. Let’s put it all together in a package we can run.
  • Add a Data Flow task to the Control Flow
  • Double click on the Data Flow task to edit it.
  • Add an “OLE DB Source” component by dragging it to the Data Flow surface.
  • Double click on the “OLE DB Source” component to edit it. The “OLE DB Connection Manager” property should already be populated for you and should be the “AdventureWorksDW” connection manager object we created from before.
  • Change the “Data access mode” to “SQL command” as this is always good practice to do. This allows us to select only the columns from the database that we need and also allows us to add a WHERE clause if necessary. This is the only practice I recommend.
  • Use the following SQL:
  • SELECT DISTINCT FirstName, LastName FROM     DimEmployee
  • Click on the “Columns” tab on the left. This populates the list of columns automatically that we want to expose to the Data Flow.
  • Click “OK”
  • Drag a “Flat File Destination” component to the Data Flow.
  • Connect the two components using the green line from the “OLE DB Source” component.
  • Double click on the “Flat File Destination” component to edit it.
  • Again, just like the previous component, the “Flat File Connection Manager” property has been automatically selected for you.
  • Click on the “Mappings” tab on the left to populate the mappings between the source and destination. Drag “FirstName” from the left to “Column 0″ on the right. Drag “LastName” from the left to “Column 1″ on the right.
  • Click “OK”
Now, we’re ready to execute the package, but let’s understand a few things first. The value currently stored in the SSIS_Configurations table currently reflect the default value you established when you created the “Directory” variable. So, to show that this works, let’s change the value of the variable in our SSIS_Configurations table to something like “C:\TEMP\”. Also, make sure that you actually have a “C:\TEMP\” directory and that you have permissions to write to it. So the package has “C:\” as the value for “Directory,” but the SSIS_Configurations table has “C:\TEMP\”. You can update this value by running an UPDATE statement in Management Studio, or by opening the table directly by right-clicking on it in Management Studio. Either way, I assume you know how to do this.
Run the package! Note that when we initially setup the “Flat File Connection Manager” object, we gave it a file name of “test_placeholder.txt”. After the package executes, you should find a file, “test.txt” in the “C:\TEMP” directory, not the “C:\” directory as you have in your variable.
What’s going on here? Well, the package configurations are applied at run-time and configure the “Directory” variable to be the value stored in the table. We then use that variable along with the “FileName” variable to dynamically configure the “ConnectionString” property of the “Flat File Connection Manager” by using expressions.


Courtesy:www.ssistalk.com


Sunday, January 9, 2011

SSIS Expression Cheat Sheet


SSIS Expression Cheat Sheet

Problems
Expression
Create a file name with today's date
Expression on the Flat File or File Connection Manager:

"C:\\Project\\MyExtract" + (DT_WSTR, 30)(DT_DBDATE)GETDATE() + ".csv"

Expression Output Example: C:\Project\MyExtract2009-03-20.csv
Use a 2 digit date
(ex. "03" for March instead of "3")
RIGHT("0" + (DT_WSTR, 2)MONTH(GETDATE()),2)


Expression Output: 03 (if the month is March)
Multiple condition if statement
In this example, the statement determines that if the ColumnName
is blank or NULL, it will be set to unknown. To make a Logical AND condition, use "&&" instead of the "||" operator.



ISNULL(ColumnName)||TRIM(ColumnName)==""?"Unknown": ColumnName
Returns the first five characters from a zip code
Derived Column Transform in the Data Flow:

SUBSTRING(ZipCodePlus4,1,5)
Remove a given character from a string
(ex. Remove "-" from a social security number)
Derived Column Transform in the Data Flow:

REPLACE(SocialSecurityNumber, "-","")
Uppercase data
Derived Column Transform in the Data Flow:

UPPER(ColumnName)
Replace NULL with another value
Derived Column Transform in the Data Flow:

ISNULL(ColumnName)?"New Value" : ColumnName
Replace blanks with NULL values
Derived Column Transform in the Data Flow:

TRIM(ColumnName)=="" ? (DT_STR, 4, 1252)NULL(DT_STR, 4, 1252) : ColumnName
Remove any non-numeric data from a column
Script Transform in the Data Flow Task with the code as follows (VB 2008):

Imports System.Text.RegularExpressions


Public Overrides Sub Input()_ProcessInputRows(ByVal Row As Input()Buffer)

If Row.ColumnName_IsNull = False Or Row.ColumnName = "" Then
Dim pattern As String = String.Empty

Dim r As Regex = Nothing

pattern = "[^0-9]"

r = New Regex(pattern, RegexOptions.Compiled)

Row.ColumnName = Regex.Replace(Row.ColumnName, pattern, "")
End If
End Sub
Convert text to proper case
(ex. 1st letter in each word is uppercase)
Script Transform with the line of partial code as follows:

Row.OutputName = StrConv(Row.InputName, VBStrConv.ProperCase)
Build dynamic SQL statement
Expression on the SQLStatementSource property of Execute SQL Task:

"SELECT Column From " + @[User::TableName] +WHERE DateFilterColumn = '" + (DT_WSTR,4)YEAR(@[User::DateTimeVar]) +
RIGHT("0" + (DT_WSTR,2)MONTH(@[User::DateTimeVar]),2) + RIGHT("0" + (DT_WSTR,2)DAY(@[User::DateTimeVar]),2) + "'"

Expression Output:SELECT Column FROM MyTable WHERE DateFilerColumn = '20060915'
Calculate beginning of the previous month
Expression on component or task:

(DT_DATE)(DT_DBDATE)DATEADD("dd",-1*(DAY(GETDATE())-1),DATEADD("month", -1, GETDATE()))
Round to the nearest two decimal mark
Expression on Derived Column Transform:

ROUND(YourNumber, 2)

Expression Output Example: 1.2600000

Common SSIS Problems and Solutions


Common SSIS Problems and Solutions

Problems
Solutions
Loop over a list of files & load each one
Tasks Required: Foreach Loop, Data Flow Task

Solution: Configure the Foreach Loop to loop over any particular directory of files. The loop should be configured to output
to a given variable. Map the given variable to a connection manager by using expressions.
Conditionally executing tasks

Solution: Double-click the precedence constraint and set the Evaluation property to Expression and Constraint.
Type the condition that you want to evaluate in the Expression box.
Pass in variables when scheduling or running a package

Solution: Use the /SET command in the DTExec command line or change the Property tab in the Package Execution Utility to have the property path like:
\Package.Variables[User::VariableName].Properties[Value]
Move and rename the file at the same time
Tasks Required: File System Task

Solution: Set the File System task to rename the file and point to the directory you'd like to move the file to. This enables you to rename and move the file in the same step.
Loop over an array of data in a table & perform a set of tasks for each row
Tasks Required: Execute SQL Task, Foreach Loop

Solution: Use an Execute SQL Task to load the array and send the data into an object variable. Loop over the variable in a Foreach Loop by use an ADO Enumerator.
Perform an incremental load of data
Tasks Required: 2 Execute SQL Tasks, Data Flow Task

Solution: Have the 1st Execute SQL Task retrieve a date from a control table of when the target table was last loaded and place that into a variable.
In the Data Flow Task, create a date range on your query using the variable. Then, update the control table using a 2nd Execute SQL Task to specify when the table was
last updated.
Perform a conditional update & insert
Components Required: Data Flow Task, Conditional Split, Lookup Transform or Merge Join, OLE DB Command Transform

Solution: Use the lookup Transform or Merge Join to detemine if the row exists on the destination and ignore a failed match. If the row yields blank
on the key, then you know the row should be inserted into the target (by Conditional Split). Otherwise, the row is a duplicate or an update. Determine if the row
is an update by comparing the source value to the target value in the Conditional Split. The update can be done by an OLE DB Command Transform or by loading the data
into a staging table.

Sunday, January 2, 2011

SQL Server Integration Services SSIS Package Configuration

Problem
As we develop, test and deploy SSIS packages, we need to specify different parameter values for things like database connection strings, file and folder paths, etc. without having to edit the individual SSIS packages to make these changes. How can we take advantage of the configuration capabilities that SSIS provides, particularly the SQL Server package configuration, to accomplish our goal? Read More..

Monday, December 20, 2010

Slowly Changing Dimension Transformation

The Slowly Changing Dimension transformation coordinates the updating and inserting of records in data warehouse dimension tables. For example, you can use this transformation to configure the transformation outputs that insert and update records in the DimProduct table of the AdventureWorksDW2008R2 OLAP database with data from the Production.Products table in the AdventureWorks2008R2 OLTP database. Read more..

Also check this article

Sunday, December 19, 2010

SQL Server Integration Services (SSIS) 15 Best Practices

Listed below are some SQL Server Integration Services (SSIS) best practices:

1.Keep it simple.

Avoid using components unnecessarily. For example:

Step 1. Declare the variable varServerDate.

Step 2. Use ExecuteSQLTask in the control flow to execute a SQL query to get the server date-time and store it in the variable

Step 3. Use the dataflow task and insert/update database with the server date-time from the variable varServerDate.

This sequence is advisable only in cases where the time difference from step 2 to step 3 really matters. If that doesn't really matter, then just use the getdate() command at step 3, as shown below:


--create table #table1 (Lap_Id int, LAP_Date datetime)

Insert into #table1 (Lap_Id, LAP_Date) values (1, getdate())2.Calling a child package multiple times from a parent with different parameter values.

When a child package is executed from a master package, the parameters that are passed from the master need to be configured in the child package. For this, you can use the ‘Parent Package Configuration’ option in the child package. But, for using the ‘Parent Package Configuration’, you need to specify the name of the ‘Parent Package Variable’ that is passed to the child package. If you want to call the same child package multiple times (each time with a different parameter value), declare the parent package variables (with the same name as given in the child package) with a scope limited to ‘Execute Package Tasks’.

SSIS allows declaring variables with the same name but the scope limited to different tasks – all inside the same package!

3.SQL job with many atomic steps.

For the SQL job that calls the SSIS packages, make multiple steps, each doing small tasks, rather than a single step doing all the tasks. In the first case, the transaction log grows too big, and if a rollback happens, it may take the full processing space of the server.

4.Avoid unnecessary typecasts.

Avoid unnecessary type casts. For example, the flat file connection manager, by default, uses the string [DT_STR] data type for all the columns. In case you want to use the actual data types, you have to manually change it. Better to change it at the source-level itself to avoid unnecessary type castings.

5.Transactions.

Usually, the ETL processes handle large volumes of data. In such a scenario, do not attempt a transaction on the whole package logic. However, SSIS supports transaction, and it is advisable to use transactions where the atomicity of the transaction is taken care of.

For example, consider a scenario where a source record is to be spitted into 25 records at the target - where either all the 25 records reach the destination or zero. In this scenario, using a transaction, we can ensure either all the 25 records reach the destination or zero.


6.Distributed transaction spanning multiple tasks.

The control flow of an SSIS package threads together various control tasks. It is possible to set a transaction that can span into multiple tasks using the same connection. To enable this, the “retainsameconnection” property of the Connection Manager should be set to “True”.

7.Limit the package names to a maximum of 100 characters.

When an SSIS package with a package name exceeding 100 chars is deployed into SQL Server, it trims the package name to 100 chars, which may cause an execution failure. So, limit the package names to a maximum of 100 characters.

8.Select * from…

Make sure that you are not passing any unnecessary columns from the source to the downstream. With the OLEDB connection manager source, using the ‘Table or View’ data access mode is equivalent to ‘SELECT * FROM ’, which will fetch all the columns. Use ‘SQL command’ to fetch only the required columns, and pass that to the downstream. At each down-stream component, filter out the unnecessary columns.

9.Sorting.

Sorting in SSIS is a time consuming operation. At the places where we know that data is coming from database tables, it’s better to perform the sorting operation at the database query itself.

10.Excel Source and 64-bit runtime.

The Excel Source or Excel Connection manager works only with the 32 bit runtime. When a package using the Excel Source is enabled for 64-bit runtime (by default, it is enabled), it will fail on the production server using the 64-bit runtime. Go to the solution property pages\debugging and set Run64BitRuntime to False.

11.On failure of a component, stop/continue the execution with the next component.


When a component fails, the property failParentonFailure can be effectively used either to stop the package execution or continue with the next component - exception - stop/continue with the next component in a sequence container. The value of the constraint connecting the components in the sequence should be set to "Completion", and the failParentonFailure property should be set to False (default).

12.Protection.

To avoid most of the package deployment error from one system to another system, set the package protection level to ‘DontSaveSenstive’.

13.Copy pasting the Script component.

Once you copy-paste a script component and execute the package, it may fail. Just open the script editor of the pasted script component, save the script, and execute the package – it will work.

14.Configuration filter – Use as a filter.

It is a best practice to use the package name as the configuration filter for all the configuration items that are specific to a package. It is especially useful when there are so many packages with package-specific configuration items. For the configuration items that are general to many packages, use a generic name.

15.Optimal use of configuration records.

Avoid the same configuration item recorded under different filter/object names. For example, if two packages are using the same connection string, you need only one configuration record. To enable this, use the same name for the connection manager in both the packages. Also, use a generic configuration filter. This is quite convenient at the time of porting from one environment to another (e.g.: from UAT to production).

Courtesy:CodeProject

SQLCat

Tuesday, September 21, 2010

Saving and Running Packages


Once created a SSIS package, you're probably ready to run it and see what it does. But first, let's look at the options for saving SSIS packages. When you work in BIDS, your SSIS package is saved as an XML file (with the extension dtsx) directly in the normal Windows file system. But that's not the only option. Packages can also be saved in the msdb database in SQL Server itself, or in a special area of the file system called the Package Store.
Storing SSIS packages in the Package Store or the msdb database makes it easier to access and manage them from SQL Server's administrative and command-line tools without needing to have any knowledge of the physical layout of the server's hard drive.
Saving Packages to Alternate Locations
To save a package to the msdb database or the Package Store, you use the File > Save Package As menu item within BIDS.

To store copies of the package you've developed, follow these steps.
  1. Select File > Save Copy of Package.dtsx As from the BIDS menus.
  2. Select SSIS Package Store as the Package Location.
  3. Select the name of your test server.
  4. Enter the package path.
  5. Click OK.
  6. Select File > Save Copy of Package.dtsx As from the BIDS menus.
  7. Select SQL Server as the Package Location.
  8. Select the name of your test server and fill in your authentication information.
  9. Enter ExportDepartments as the package path.
  10. Click OK.
Running a Package
You can run the final package from either BIDS or SQL Server Management Studio. When you're developing a package, it's convenient to run it directly from BIDS. When the package has been deployed to a production server (and saved to the msdb database or the Package Store) you'll probably want to run it from SQL Server Management Studio.
SQL Server also includes a command-line utility, dtsexec, that lets you run packages from batch files.
Running a Package from BIDS
With the package open in BIDS, you can run it using the standard Visual Studio tools for running a project. Choose any of these options:
  • Right-click the package in Solution Explorer and select Execute Package.
  • Click the Start Debugging toolbar button.
  • Press F5.
To run the package that you have loaded in BIDS, follow these steps:

  1. Click the Start Debugging toolbar button. SSIS will execute the package, highlighting the steps in the package as they are completed. You can select any tab to watch what's going on. For example, if you select the Control Flow tab, you'll see tasks highlighted, as shown in Figure. 

     2. When the package finishes executing, click the hyperlink underneath the Connection    Managers pane to stop the debugger.
    3. Click the Execution Results tab to see detailed information on the package, as shown in Figure.

All of the events you see in the Execution Results pane are things that you can create event handlers to react to within the package. As you can see, DTS issues a quite a number of events, from progress events to warnings about extra columns of data that we retrieved but never used.
Running a Package from SQL Server Management Studio
To run a package from SQL Server Management Studio, you need to connect Object Browser to SSIS.
Try It!
  1. In SQL Server Management Studio, click the Connect button at the top of the Object Explorer window.
  2. Select Integration Services.
  3. Choose the server with Integration Services installed and click Connect. This will add an Integration Services node at the bottom of Object Explorer.
  4. Expand the Stored Packages node. You'll see that you can drill down into the File System node to find packages in the Package Store, or the MSDB node to find packages stored in the msdb database.
  5. Expand the File System node.
  6. Right-click on the package and select Run Package. This will open the Execute Package utility, shown in Figure .



  1. Click Execute.
  2. Click Close twice to dismiss the progress dialog box and the Execute Package Utility.
  3. Browse for the inserted data
  1. Click the Execute toolbar button to verify that the package was run. You should see one entry for when the package was run from BIDS and one from when you ran it from SQL Server Management Studio.

Monday, August 30, 2010

SSIS Multicast Transformation vs Conditional Split Transformation

In a data warehousing scenario, it's not rare to replicate data of a source table to multiple destination tables, sometimes it's even required to distribute data of a source table to two or more tables depending on some condition. For example splitting data based on location etc. So how we can achieve this with SSIS? SSIS provides several built-in transformation tasks to achieve these kinds of , for details click here.

Working with SQL Server Service Broker (SSBS) When the Initiator and Target are on the Same Database

Introduction

In my previous article, I introduced you SQL Server Service Broker, what it is, how it works, what its different components are and how they are related to each other. Now it's time to roll up our sleeves and write some SSBS applications. In this article, I will be creating an application in which Initiator and Target both are in the same database. In the next couple of articles, I will be helping you to write SSBS applications if the Initiator and Target are not in the same database.

Problem Statement

There are two applications, one is Order application and the other one is Inventory application. Before accepting any order from the users, the Order application needs to make sure that the product, which is being ordered, is available in the store but for that, the Order application does not want to wait. The Order application will request to check the product stock status asynchronously and will continue doing its other work. On the other side, the Inventory application will listen to the request coming into the queue, process it and respond back to the Order application, again asynchronously.

All in one database

Depending on the arrangement of Initiator and Target, the architecture can be grouped into three categories:

                •Initiator and Target in same database

                •Initiator in one database and Target in another database on same instance

              •Initiator and Target in separate instances

In this example, I will demonstrate how to create an SSBS application if both Initiator and Target are in same database.



Though we can enable Service Broker for an existing database and create SSBS objects in it, for simplicity I will be creating a new database for this demonstration.

USE master;
GO
--Create a database for this learning session, it will help you to do 
--clean up easily, you can create SSBS objects in any existing database
--also but you need to drop all objects individually if you want to do
--clean up of these objects than dropping a single database
IF EXISTS(SELECT COUNT(1) FROM sys.databases WHERE name = 'SSBSLearning')
 DROP DATABASE SSBSLearning
GO
CREATE DATABASE SSBSLearning
GO
 
--By default a database will have service broker enabled, which you can verify 
--with is_broker_enabled column of the below resultset
SELECT name, service_broker_guid, is_broker_enabled, is_honor_broker_priority_on 
FROM sys.databases WHERE name = 'SSBSLearning'
 
--If your database is not enabled for Service Broker becuase you have 
--changed the default setting in Model database, even then you can enable
--service broker for a database with this statement
ALTER DATABASE SSBSLearning
      SET ENABLE_BROKER;
      --WITH ROLLBACK IMMEDIATE
GO
----To disable service broker for a database
--ALTER DATABASE SSBSLearning
--      SET DISABLE_BROKER;
--GO

Once you have created a database or enabled the Service Broker for an existing database, you need to create the service broker objects; first Message Types then Contracts, which will use the created message types. Then you need to create Queues and finally you would be required to create Services, which are nothing but endpoints that sit on top of queues to send and receive messages.

USE SSBSLearning;
GO
--Create message types which will allow valid xml messages to be sent
--and received, SSBS validates whether a message is well formed XML 
--or not by loading it into XML parser 
CREATE MESSAGE TYPE
       [//SSBSLearning/ProductStockStatusCheckRequest]
       VALIDATION = WELL_FORMED_XML;
CREATE MESSAGE TYPE
       [//SSBSLearning/ProductStockStatusCheckResponse]
       VALIDATION = WELL_FORMED_XML;
GO
 
--Create a contract which will be used by Service to validate
--what message types are allowed for Initiator and for Target.
--As because communication starts from Initiator hence 
--SENT BY INITIATOR or SENT BY ANY is mandatory
CREATE CONTRACT [//SSBSLearning/ProductStockStatusCheckContract]
      ([//SSBSLearning/ProductStockStatusCheckRequest]
       SENT BY INITIATOR,
       [//SSBSLearning/ProductStockStatusCheckResponse]
       SENT BY TARGET
      );
GO
 
--Create a queue which is an internal physical table to hold 
--the messages passed to the service, by default it will be 
--created in default file group, if you want to create it in 
--another file group you need to specify the ON clause with 
--this statement. You can use SELECT statement to query this 
--queue or special table but you can not use other DML statement
--like INSERT, UPDATE and DELETE. You need to use SEND and RECEIVE
--commands to send messages to queue and receive from it
CREATE QUEUE dbo.SSBSLearningTargetQueue;
GO
 
--Create a service, which is a logical endpoint which sits on top 
--of a queue on which either message is sent or received. With 
--Service creation you all specify the contract which will be
--used to validate message sent on that service
CREATE SERVICE
       [//SSBSLearning/ProductStockStatusCheck/TargetService]
       ON QUEUE dbo.SSBSLearningTargetQueue
       ([//SSBSLearning/ProductStockStatusCheckContract]);
GO
 
--A Target can also send messages back to Initiator and hence
--you can create a queue for Initiator also
CREATE QUEUE dbo.SSBSLearningInitiatorQueue;
GO
 
--Likewsie you would need to create a service which will sit 
--on top of Initiator queue and used by Target to send messages
--back to Initiator
CREATE SERVICE
       [//SSBSLearning/ProductStockStatusCheck/InitiatorService]
       ON QUEUE dbo.SSBSLearningInitiatorQueue;
GO

Once you are done creating all of the required SSBS objects for both Initiator (Order application) and Target (Inventory application), you will be sending messages between Initiator and Target. A dialog or conversation is always started by Initiator and therefore Initiator will be sending the request message first (to check the inventory stock for product as mentioned in problem statement) to the Target something
like this.

--To send message, first you need to initiate a dialog with 
--BEGIN DIALOG command and specify the Initiator and Target
--services which will be talking in this dialog conversation
DECLARE @SSBSInitiatorDialogHandle UNIQUEIDENTIFIER;
DECLARE @RequestMessage XML;
BEGIN TRANSACTION;
 BEGIN DIALOG @SSBSInitiatorDialogHandle
   FROM SERVICE
    [//SSBSLearning/ProductStockStatusCheck/InitiatorService]
   TO SERVICE
    N'//SSBSLearning/ProductStockStatusCheck/TargetService'
   ON CONTRACT
    [//SSBSLearning/ProductStockStatusCheckContract]
   WITH ENCRYPTION = OFF;
 SELECT @RequestMessage =
     N'<Request>
    <ProductID>316</ProductID>
    <LocationID>10</LocationID>
   </Request&g';
 --To send message you use SEND command and specify the dialog
 --handle which you got above after initiating a dialog  
 SEND ON CONVERSATION @SSBSInitiatorDialogHandle
   MESSAGE TYPE 
   [//SSBSLearning/ProductStockStatusCheckRequest]
   (@RequestMessage);
 SELECT @RequestMessage AS RequestMessageSent;
COMMIT TRANSACTION;
GO

The request message sent using the above statements is stored in the Target queue until it is processed. You can query the Target queue using a SELECT statement.

--You can query the Target queue with SELECT statement
SELECT * FROM dbo.SSBSLearningTargetQueue;
GO
--If in case message cannot be put into Target queue becuase
--Target queue is not enabled or because of any other reasons
--it will be temporarily put into transmission queue until 
--its delivery to Target queue
SELECT * FROM sys.transmission_queue
GO

Next the Target (Inventory application) will pick up the request messages from its queue, process it and respond back to the Initiator (Order application) with the product inventory status; at the end it will end the dialog conversation, which Initiator initiated as it is no longer required.

--To retrieve a message from a queue you use RECEIVE command,
--With every message you also get dialog handle which you can 
--use to reply back to sender of the message
DECLARE @SSBSTargetDialogHandle UNIQUEIDENTIFIER;
DECLARE @RecvdRequestMessage XML;
DECLARE @RecvdRequestMessageTypeName sysname;
BEGIN TRANSACTION;
 --WAITFOR command is used to wait for messages to arrive
 --on the queue, TIMEOUT is specified in miliseconds
 WAITFOR
 ( RECEIVE TOP(1)
  @SSBSTargetDialogHandle = conversation_handle,
  @RecvdRequestMessage = CONVERT(XML, message_body),
  @RecvdRequestMessageTypeName = message_type_name
   FROM dbo.SSBSLearningTargetQueue
 ), TIMEOUT 1000;
 
 SELECT @RecvdRequestMessage AS RequestMessageReceived;
 --If the message type is request from Initiator, process the request.
 IF @RecvdRequestMessageTypeName = N'//SSBSLearning/ProductStockStatusCheckRequest'
 BEGIN
   DECLARE @ReplyMessage NVARCHAR(max);
   DECLARE @Quantity smallint
   SELECT @Quantity = Quantity 
   FROM AdventureWorks.Production.ProductInventory
   WHERE ProductID = @RecvdRequestMessage.value('(/Request/ProductID)[1]', 'int')  
   AND LocationID = @RecvdRequestMessage.value('(/Request/LocationID)[1]', 'int') 
   SELECT @ReplyMessage = 
   N'<Reply>
   <Quantity>' + CONVERT(VARCHAR(10), @Quantity) + '</Quantity>
   </Reply>';
   --To send message back to sender you again use SEND command and specify the dialog
   --handle which you got above while retrieving the message from the queue
   SEND ON CONVERSATION @SSBSTargetDialogHandle
     MESSAGE TYPE 
     [//SSBSLearning/ProductStockStatusCheckResponse]
     (@ReplyMessage);
   --To end a dialog you use END CONVERSATION command, here the dialog
   --is being ended by Target, and then a message of 
   -- http://schemas.microsoft.com/SQL/ServiceBroker/EndDialog message type 
   --will be sent to Initiator
   END CONVERSATION @SSBSTargetDialogHandle;
 END
 SELECT @ReplyMessage AS ReplyMessageSent;
COMMIT TRANSACTION;
GO

The response message (which Target sends) comes and resides in the Initiator queue, which you can see by querying the Initiator queue with a SELECT statement as you did for Target queue above.

--You can query the Initiator queue with SELECT statement
--Here you will see two records, one is the response and another
--one is the end dialog intimation with NULL in its message_body column
SELECT * FROM dbo.SSBSLearningInitiatorQueue
GO
--Again if there is any error during transmission for messages from Target to 
--Initiator, the messages will be temporarily put into transmission queue until 
--its delivery to Initiator queue
SELECT * FROM sys.transmission_queue
GO

At this point, the Initiator (Order application) can retrieve the product status response message from its
queue, which was written by Target (Inventory application). Please note, if you use the above statement to see the Initiator queue, you will see two records; the first one is the response message from the Target and second one message is for dialog end, which was ended by Target after sending the response. To end a
dialog successfully, it has to be ended by both Target and Initiator. Hence, you need to execute the script given below twice; the first time to process the response and the second time to end the conversation as it is already ended by Target.

--At this point the Initiator queue will hold two records, first
--one is a response message for the request and another one is for 
--intimation that dialog has been ended by the Target.
--You need to execute below piece of code twice to retrive both the 
--records from Initiator queue, if the message is of type 
-- http://schemas.microsoft.com/SQL/ServiceBroker/EndDialog the dialog will
--be ended by Intiator also or else response message will be processed. 
DECLARE @RecvSSBSInitiatorDialogHandle UNIQUEIDENTIFIER;
DECLARE @RecvReplyMessage NVARCHAR(100);
DECLARE @RecvMessageType SYSNAME
BEGIN TRANSACTION;
 WAITFOR
 ( RECEIVE TOP(1)
  @RecvSSBSInitiatorDialogHandle = conversation_handle,
  @RecvReplyMessage = message_body,
  @RecvMessageType = message_type_name
   FROM dbo.SSBSLearningInitiatorQueue
 ), TIMEOUT 1000;
 --If the message is of type http://schemas.microsoft.com/SQL/ServiceBroker/EndDialog 
 --the dialog will be ended by Intiator also or else response message will be processed.
 IF (@RecvMessageType = 'http://schemas.microsoft.com/SQL/ServiceBroker/EndDialog')
  END CONVERSATION @RecvSSBSInitiatorDialogHandle;
 ELSE
  SELECT @RecvReplyMessage AS ReplyMessageRecived;
COMMIT TRANSACTION;
GO
 

Once you are done with the testing and want to drop all the objects which you created, you can use the clean-up scripts below.

--Cleanup code to drop SSBS objects individually which 
--you created above, notice the order of dropping the objects,
--its reverse of the order in which you created the objects
DROP SERVICE [//SSBSLearning/ProductStockStatusCheck/InitiatorService]
DROP SERVICE [//SSBSLearning/ProductStockStatusCheck/TargetService]
DROP QUEUE dbo.SSBSLearningInitiatorQueue
DROP QUEUE dbo.SSBSLearningTargetQueue
DROP CONTRACT [//SSBSLearning/ProductStockStatusCheckContract]
DROP MESSAGE TYPE [//SSBSLearning/ProductStockStatusCheckRequest]
DROP MESSAGE TYPE [//SSBSLearning/ProductStockStatusCheckResponse]
GO
--Drop the database which you created above
IF EXISTS(SELECT COUNT(1) FROM sys.databases WHERE name = 'SSBSLearning')
 DROP DATABASE SSBSLearning
GO

Note: Needless to say, you must learn and do thorough testing of your SSBS application (or the scripts demonstrated above) first on your development box before going to production environment.

SQL Server Service Broker - An Introduction

SQL Server Service Broker (SSBS), introduced with SQL Server 2005 and enhanced in SQL Server 2008, allows you to write queuing/message based applications within the database itself. This article discusses SSBS' important features and how to use it in different scenarios.

Introduction


SQL Server Service Broker (SSBS) is a new architecture (introduced with SQL Server 2005 and enhanced further in SQL Server 2008) which allows you to write asynchronous, decoupled, distributed, persistent, reliable, scalable and secure queuing/message based applications within the database itself. This integration of queuing and message based infrastructure into the database provides several advantages from other message/queuing based applications out of the database. For example, SSBS provides simplified administration, inbuilt security, improved performance, transaction support without making it distributed (more details later), simplified recovery process as queued data are backed up as part of the normal database back up etc. SSBS in itself is a very big topic, requiring several hundred pages to explain. Therefore, I will be talking about its important features; how to use it in different scenarios and what are the improvements in SQL Server 2008 for SSBS across several articles.

This introductory article on SSBS discusses on a very basic level, how it differs from MSMQ, looks at SSBS architecture and its different components.

How SSBS differs from MSMQ/Other message based technology


Similar to SSBS, Microsoft Message Queuing (MSMQ) is another technology that enables applications to send and read messages from queues; it comes as a part of the Windows OS. However, SSBS outweighs the features and benefits that MSMQ provides. For example, in comparison with MSMQ, SSBS provides the following:


          •Faster transactional message processing

          •Reliability and enhanced recoverability

          •Optimize the performance

          •Inbuilt Security

          •Parallel processing

          •Reduced management and maintenance overhead

MSMQ has been in use a long time and is a well-tested technology. Even though MSMQ is the preferred technology for writing queuing and message based applications and provides several advantages over other alternatives, in comparison to SSBS, SSBS outweighs MSMQ in several ways in an environment where a SQL Server instance is already available.

Advantages of SSBS over MSMQ/other queuing and message based technology

       •SSBS is a part of the database engine itself and is more efficient for data-intensive queuing operations. It has improved performance, efficient and faster transactional message processing because transaction contexts are managed by SQL Server internally, whereas MSMQ requires Distributed Transaction Coordinator (DTC) to coordinate transaction context between MSMQ and SQL Server. Remember DTC uses two-phase commit protocol.

         •Service Broker queues are maintained in the database as database objects and are backed up during normal database backup processes. This provides reliability and enhanced recoverability. MSMQ stores messages in the memory when you optimize it for faster performance. This is fine as long as there is no failure, but if there is any failure or you restart MSMQ service, the stored messages will be gone.

        •SSBS optimizes the performance by directly writing messages to the receiver queue (bypassing the sender queue) if the receiver queue is in same database or another database on the same server.

       •SSBS handles sending and receiving messages internally by database and SSBS commands are executed over the same connection to the database (no need of separate connection and therefore no need of DTC as would be required when it is external to database). This ensures the access permissions of the senders and receivers are checked by the database engine itself. Having a single identity check for messaging and database it provides the inbuilt security, which is easier to configure and maintain in one place.

        •SSBS achieves parallel processing by letting us define the number of simultaneous queue readers. Each reader can process the messages of a particular conversation group. To ensure messages of a particular conversation group are being processed by a single reader it uses a new kind of lock, called conversation group lock.

        •With SSBS, you store both logic and data in the database itself, so chances of getting both out of sync is minimized to a great extent when you restore your system after any failures.

MSMQ uses TCP/IP or HTTP transport protocols whereas SSBS supports only TCP/IP so if you want your messages to communicate over HTTP, MSMQ would be your choice.

Service Broker - Architecture

          In SSBS terminology, the Sender of the message is called Initiator and the Receiver of the message is called Target. Depending on the placement of Initiator and Target the SSBS application's architectures can be categorized in three different scenarios.

         A.Initiator and Target in the same database in the same SQL Server instance.

        B.Initiator and Target in different databases but still at the same SQL Server instance.

        C.Initiator in a database at one SQL Server instance whereas Target in another database at another    SQL Server instance.

          The behavior in scenario A and B above is almost same. In these cases, SSBS optimizes performance by writing messages directly to the Target Queue. If, while writing message to the Target Queue, it encounters any problem (for example the Target service is not available, Target Queue is disabled, etc.) it keeps that message in the sys.transmission_queue table temporarily so that it can push the message to Target Queue once it is available. In the image below, you can see there are two databases, Initiator and Target, on a SQL Server instance. When Initiator sends (by using SEND T-SQL command) a message, the message is directly written to Target Queue from where the Target reads (by using RECEIVE T-SQL command) it. If the Target sends a response back to the Initiator, the message is directly written to Initiator Queue.

             The behavior in the third scenario, scenario C, becomes a bit complex and interesting too. In this case, messages travel from one server to another and the network is involved, hence SSBS ensures successful delivery of the message by storing messages temporarily in the sys.transmission_queue temporary queue at Initiator and removing it from there only if it receives receipt acknowledgement from the Target. This means as long as messages are in transit they will reside in the sys.transmission_queue queue. This is what has been depicted in the image below. You can see there is a queue at database level and there is sys.transmission_queue temporary queue at instance wide.

When an application issues the SEND command to send a message, that message is stored in sys.transmission_queue queue at Initiator, then SSBS sends that message over the network to the Target and at same time, it marks the status of this message as waiting for acknowledgement from the Target. On the Target, when the message is received successfully in the Target queue, it sends the acknowledgement back to Initiator. At this point only, the message is deleted from sys.transmission_queue queue at the Initiator. If the Target issues a SEND command to send a response message back to the sender, the response message goes into the sys.transmission_queue queue at the Target, again SSBS sends that response message over the network to the Initiator and at same time, it marks the status of this response message as waiting for acknowledgement from the Initiator. On the Initiator, when the response message is received successfully in the Initiator queue, it sends the acknowledgement back to the Target. At this point only, the message is deleted from the sys.transmission_queue queue at the Target.

Please note, the sys.transmission_queue temporary queue is not specific to a database but rather is only for each instance.

SSIS-Parallel Processing

Parallel execution improves performance on computers that have multiple physical or logical processors. To support parallel execution of different tasks in the package, SSIS uses two properties: MaxConcurrentExecutables and EngineThreads.



MaxConcurrentExcecutables Property

The MaxConcurrentExecutables property is a property of the package. This property defines how many tasks can run simultaneously; by specifying the maximum number of SSIS threads that can execute in parallel per package. The default value is -1, which equates to the number of physical or logical processors plus 2. Using a package which calls another package, in this example MaxConcurrentExecutables has its default value set as -1 and the server running the package has 8 processors, all 10 tasks (taking the Execute Package task in the example, though it applies in the same way to other tasks as well) are executed in parallel, as shown below:

If MaxConcurrentExecutables was changed to 4 in the above package and run it on the same server, then only 4 tasks will runn in parallel at a time (Note the image below shows tasks are executed in a batch of 4, once 4 tasks are executed another batch of 4 tasks will be executed)



Note:

If SSIS runs on a dedicated server and there are a lot of operations that run in parallel, it can be beneficial to increase this setting if some of the operations (threads) do a lot of waiting for external systems to reply. However if there's no dedicated SSIS machine and the data integration application runs alongside several other applications, reducing this setting should avoid resource conflicts.

EngineThreads Property

The EngineThreads property is a property of each Data Flow task. This property defines how many threads the data flow engine can create and run in parallel. The EngineThreads property applies equally to both the source threads that the data flow engine creates for sources and the worker threads that the engine creates for transformations and destinations. Therefore, setting EngineThreads to 10 means that the engine can create up to ten source threads and up to ten worker threads.

Note:

A source thread pulls the data from the data source whereas worker thread performs transformation and upload data into the destination.



To understand how this property works, consider the sample package with three Data Flow tasks. Each Data Flow task contains ten source-to-destination execution trees. If the EngineThreads are set to 10 on each Data Flow task, all 30 execution trees can potentially run simultaneously (considering all three data flow tasks run in parallel because of MaxConcurrentExecutables property of the package has either -1 or 3).

As discussed above, the execution tree is a unit of work for a worker thread in the SSIS process. A worker thread may execute one or more execution trees, but each transformation listed in the execution tree will be executed on the same thread. A thread will process one buffer at a time, executing it against all transforms in the execution tree before working on the next buffer in the flow, at which point it would pass the current buffer to another thread executing another execution tree and it would pull a new data buffer from its buffer list which was queued from an upstream component (either a data source or the last asynchronous transform before this execution tree started).

Because an execution tree is tied to a thread, it is often recommended to increase the EngineThreads setting to equal or greater than the number of execution trees listed in the PipelineExecutionTrees log event.

The default is 5 in SQL Server 2005 and 10 in SQL Server 2008, with a minimum value of 2. However, the engine will not use more threads than it needs, regardless of the value of this property. The engine may also use more threads than specified in this property, if necessary to avoid concurrency issues.

However, the general rule is to not run more threads in parallel than the number of available processors. Running more threads than the number of available processors can hinder performance because of the frequent context-switching between threads.



Note:

Parallelism is a double-edge sword, on one hand a properly designed package noticeably improves performance by parallel running several tasks of the package or several execution trees of data flow; however a poorly designed package can severely impact performance if the number of threads exceed the number of available processors, throughput may be harmed due to an excessive amount of context switching. Therefore it's recommended that testing takes place considering available system resources or environment, loaded on the server, if the server is only ETL server or it is a database server as well.