Distributing Data for Availability and Scalability

(1)

Distributing Data for Availability and

Scalability

Distributing Data for Availability and

Scalability

Don Vilen Don Vilen

Program Manager Program Manager

SQL Server SQL Server

Microsoft Corporation Microsoft Corporation

(2)

Agenda Agenda

 Replication Technology PositioningReplication Technology Positioning

 SQL Server Replication Customer SQL Server Replication Customer Scenarios

Scenarios

 Upgrade StrategyUpgrade Strategy

 Key Improvements in SQL Server Key Improvements in SQL Server 2005 Replication

2005 Replication

(3)

SQL Server Replication SQL Server Replication

A Spectrum of Solutions A Spectrum of Solutions

Merge Merge Replication Replication

Queued Queued Updating Updating Subscribers Subscribers Automatic conflict detection &

Automatic conflict detection &

resolution resolution

Linked Linked Servers and Servers and

Microsoft Microsoft Distributed Distributed Transaction Transaction Coordinator Coordinator Transactional

Transactional Replication Replication Peer-to-Peer Peer-to-Peer

Immediate Immediate Updating Updating Subscribers Subscribers

Conflicts are avoided through partitioning or two- Conflicts are avoided through partitioning or two-

phase commit phase commit Increasing autonomy

Increasing autonomy Increasing consistencyIncreasing consistency

(4)

Replication Technologies Replication Technologies

Server Scenarios Server Scenarios

 Transactional ReplicationTransactional Replication

 Best choice continuously synchronizing Best choice continuously synchronizing databases to support scaling out query databases to support scaling out query

workloads workloads

 SQL Server Integration Services (was DTS)SQL Server Integration Services (was DTS)

 Best choice for performing transformations on Best choice for performing transformations on data during transfer between databases

data during transfer between databases

 Database MirroringDatabase Mirroring

 Best choice for providing database high Best choice for providing database high availability

availability

 Disk / Device MirroringDisk / Device Mirroring

 Third-party solutionsThird-party solutions

 Ensure it meets SQL Server I/O Basics Ensure it meets SQL Server I/O Basics requirements

requirements

(5)

Mobile Synchronization Mobile Synchronization

Comparison

^Replicati^Replicati^Merge^Merge

onon

Offline Offline Datasets

Datasets Web Web Services Services

Incremental Changes Incremental Changes

Conflict detection and resolution Conflict detection and resolution Automatically initialize schema Automatically initialize schema and data

and data

Supports large datasets Supports large datasets

Query processor locally available Query processor locally available Unique IDs across remote sites Unique IDs across remote sites Database constraints locally Database constraints locally maintained

maintained

Supports heterogeneous Supports heterogeneous databases

databases

Does not affect database schema Does not affect database schema Deploy application and db

Deploy application and db together

together

(6)

Agenda Agenda

Scenarios

2005 Replication

(7)

Customer Scenarios Customer Scenarios

 Offloading reporting and Data Offloading reporting and Data Warehousing

Warehousing

 Database scale out for websitesDatabase scale out for websites

 Empowering mobile usersEmpowering mobile users

 Enabling Point of Sale applicationsEnabling Point of Sale applications

 Consolidating and distributing Consolidating and distributing regional data

regional data

 Integrating heterogeneous dataIntegrating heterogeneous data

(8)

Reporting and Data Reporting and Data

Warehousing Warehousing

 Reporting databases Reporting databases are continuously

are continuously

synchronized in near synchronized in near real-time with one or real-time with one or

more source OLTP more source OLTP

databases databases

 Reporting and Staging Reporting and Staging user queries are

user queries are offloaded from the offloaded from the

OLTP databases OLTP databases

 Reporting data can be Reporting data can be a subset of source

a subset of source tables and columns tables and columns

and also indexed and also indexed

differently differently

 Read-only replicasRead-only replicas

(9)

Database Scale Out Database Scale Out

 Queries scaled out Queries scaled out (often

(often

geographically) geographically)

similar to reporting similar to reporting

cases cases

 But databases But databases replicate

replicate

reciprocally and are reciprocally and are

writable writable

 Redundancy Redundancy provides fault provides fault tolerance and tolerance and

lowers lowers

maintenance maintenance

downtime downtime

 Online upgrades Online upgrades possible

possible

London

Chicago

Tokyo

(10)

Mobile CRM / SFA Mobile CRM / SFA

Customer Relationship Customer Relationship

Management Management

Sales Force Automation Sales Force Automation

 Key Traits

 Subset of data from server

 Conflicts

 Data

realignment

 Business Logic

(11)

Point of Sale Point of Sale

 Key Traits

 High latency low bandwidth network

 Need low cost software and administratio n

 Sync at

scheduled times

 Upload

orders then remove

(12)

Regional Replication Regional Replication

 Central Central

headquarters data headquarters data replicated to 1200 replicated to 1200

stores stores

 Each store also Each store also

replicates between replicates between

two databases to two databases to provide a backup provide a backup

in case one fails in case one fails

 Database Database

administrators are administrators are

centralized centralized

(13)

Regional Replication Regional Replication

 Each ship contains Each ship contains a SQL Server

a SQL Server database

database

 When the ship When the ship

docks, it replicates docks, it replicates

to the to the

headquarters’

database database

 Replication was Replication was customized to customized to

perform additional perform additional

processing when processing when

synchronizing synchronizing

SQL Server SQL Server

Customized Customized Replication Replication

Logic Logic

Transactional Replication

Headquarters Headquarters

(14)

Oracle Replication Oracle Replication

 SQL Server 2005 SQL Server 2005 Transactional

Transactional Replication Replication

 SQL Server SQL Server

synchronized with synchronized with

Oracle production line Oracle production line

system in near real- system in near real-

time time

 Used to monitor Used to monitor manufacturing manufacturing

 Reporting: Quality Reporting: Quality control, status, and control, status, and

“what if” queries

spread across multiple spread across multiple

databases databases

Manufacturing System Manufacturing System

Oracle Oracle

10G10G

SQL Server 2005 SQL Server 2005

Transactional Replication

(15)

Top SQL Server Customers Top SQL Server Customers

Transactional Replication Transactional Replication

 Runs many core trading applications on SQL Runs many core trading applications on SQL Server

Server

 Replicates data to integrated multiple trading Replicates data to integrated multiple trading systems

systems

 15+ TB data in multiple systems, multiple 15+ TB data in multiple systems, multiple hospitals

hospitals

 Replicates to meet HA requirementsReplicates to meet HA requirements

 Leading discount retailer, SQL Server in 1200 Leading discount retailer, SQL Server in 1200 stores

stores

 Replicates data up and down between stores & Replicates data up and down between stores &

HQHQ

[Click logo for case-study]

(16)

Agenda Agenda

Scenarios

2005 Replication

(17)

Upgrade: Replication Upgrade: Replication

 SQL Server 2005 Express Edition does SQL Server 2005 Express Edition does not support Merge publishing

not support Merge publishing

 SQL Server 2000 MSDE did support itSQL Server 2000 MSDE did support it

 SQL Server Express requires custom SQL Server Express requires custom code or Windows Sync Manager to code or Windows Sync Manager to

synchronize synchronize

 If you run agents as DBO, now If you run agents as DBO, now requires a

requires a job_loginjob_login and and job_passwordjob_password to be added to replication scripts

to be added to replication scripts

(18)

Upgrade: Merge Upgrade: Merge

Replication Replication

 Snapshot must be executed after Snapshot must be executed after setup

setup

 Upgrade orderUpgrade order

 Distributor, publisher, subscriber Distributor, publisher, subscriber

 Applications must be built on Visual Applications must be built on Visual Studio 2005 to use programming APIs Studio 2005 to use programming APIs

 SQL Server 2005 subscribers SQL Server 2005 subscribers outperform previous versions outperform previous versions

(19)

Agenda Agenda

Scenarios

2005 Replication

(20)

Database Schema Database Schema

Changes Changes

 One of the key problems when One of the key problems when

replicating data is accommodating replicating data is accommodating

schema changes without disrupting schema changes without disrupting

replication replication

 SQL Server 2005 supports adding, SQL Server 2005 supports adding, dropping, and altering columns

dropping, and altering columns without stopping replication

without stopping replication

 Alter table alter column ( foo int )Alter table alter column ( foo int )

 Tables and other supported objects Tables and other supported objects can also be added or dropped

can also be added or dropped

(21)

Database Scale-Out Database Scale-Out

Requirements Requirements

 Maximize website uptimeMaximize website uptime

 Database upgrades or failures shouldn’t bring Database upgrades or failures shouldn’t bring down the application system

down the application system

 Database upgrades should be done without Database upgrades should be done without outages

outages

 Scale query workloads beyond what’s Scale query workloads beyond what’s possible with a single database

possible with a single database

 Must be possible to add or remove databases Must be possible to add or remove databases without an outage while also maintaining

without an outage while also maintaining consistency

consistency

 SQL Server 2005 Peer-to-Peer Replication SQL Server 2005 Peer-to-Peer Replication provides this capability

provides this capability

 Workbench wizard allows easy replication setupWorkbench wizard allows easy replication setup

(22)

Distribution Agent

Dist DB

Logreader Agent

Dist DB

London Chicago

Tokyo

Peer-to-Peer Topology Peer-to-Peer Topology

Transactional Replication

(23)

Replicating Large Replicating Large

Databases Databases

 The key problem with replicating The key problem with replicating large databases is initializing the large databases is initializing the

subscriber subscriber

 This can be time-consuming and requires This can be time-consuming and requires downtime to ensure consistency of the

downtime to ensure consistency of the replicated database

replicated database

 SQL Server 2005 has support for SQL Server 2005 has support for

using an existing database backup to using an existing database backup to

perform the initial synchronization perform the initial synchronization

 Initialization with a backupInitialization with a backup

 Changes that have occurred on the Changes that have occurred on the source system after the backup are source system after the backup are

tracked and synchronized tracked and synchronized

automatically automatically

(24)

Synchronizing Data Over the Synchronizing Data Over the Web Web

 Key customer driversKey customer drivers

 Wireless increasingly pervasiveWireless increasingly pervasive

 Virtual Private Network (VPN) too Virtual Private Network (VPN) too expensive or complicated for

expensive or complicated for organizations

organizations

-InternetUrl -InternetUrl

https://myserver.mydomain.com https://myserver.mydomain.com

Publisher/Distributor Publisher/Distributor

Firewall Firewall

Internet Internet Information Information Services (IIS) Services (IIS) Subscriber

Subscriber

Replisapi.dll Replisapi.dll

(25)

Logical Records Logical Records

 ProblemProblem

 Merge replication tracks changes row by Merge replication tracks changes row by rowrow

 An actual order is represented in the An actual order is represented in the Customers, Orders, OrderDetails tables Customers, Orders, OrderDetails tables

 Disconnection during sync results in Disconnection during sync results in incomplete order on server

incomplete order on server

 SolutionSolution

 Logical records create a logical Logical records create a logical consistency unit between tables consistency unit between tables

 All three changes are sent or nothing at All three changes are sent or nothing at allall

(26)

Business Logic Business Logic

 APIs to allow custom programming APIs to allow custom programming during the synchronization of

during the synchronization of changes

changes

 Allows developer to take custom Allows developer to take custom actions on different states of data actions on different states of data

 Publisher insert, update, deletePublisher insert, update, delete

 Subscriber insert, update deleteSubscriber insert, update delete

 Business Logic is registered on a per Business Logic is registered on a per article basis

article basis

(27)

Merge Performance Merge Performance

Comparison

(28)

Scalability Comparison

(29)

Monitoring Monitoring

Goals Goals

 Provide answers to common Provide answers to common questions:

questions:

 Why is the system slow?Why is the system slow?

 How long until it catches up or finishes?How long until it catches up or finishes?

 Where are the potential problems?Where are the potential problems?

(30)

Oracle Publishing Oracle Publishing

 Designed specifically for Oracle PublishersDesigned specifically for Oracle Publishers

 v8+ on any operating system v8+ on any operating system

 Administered like SQL Server, from SQL Administered like SQL Server, from SQL Server

Server

 No Oracle-side software install necessaryNo Oracle-side software install necessary

 Requires minimal knowledge of Oracle Requires minimal knowledge of Oracle

 Leverages existing SQL Server skillsLeverages existing SQL Server skills

 Standard Transactional and Snapshot Standard Transactional and Snapshot Publications

Publications

20052005

Distributor Distributor

Subscribers Subscribers

Transactional Transactional

Replication Replication

ORACLE

(31)

Many Other New Features Many Other New Features

 Transactional tracer tokenTransactional tracer token

 Merge Article typesMerge Article types

 Tables, views, stored procedures, user-Tables, views, stored procedures, user- defined functions, and stored procedure defined functions, and stored procedure

execution execution

 Resumable snapshotResumable snapshot

 Parallelism and contention Parallelism and contention improvements

improvements

 Wizards simplifiedWizards simplified

 Replication Management Objects Replication Management Objects ((RMORMO) managed replication APIs) managed replication APIs

 SecuritySecurity

(32)

Summary Summary

Scenarios

2005 Replication

(33)

For More Information For More Information

 Books Online Replication SectionBooks Online Replication Section

 Merge ReplicationMerge Replication

 White PaperWhite Paper

 Oracle Oracle

 Oracle Publishing Quick Start WhitepaperOracle Publishing Quick Start Whitepaper

 SQL Server 2005 Oracle Replication SQL Server 2005 Oracle Replication Overview

Overview

 Peer-to-Peer Peer-to-Peer

 Books OnlineBooks Online

(34)

This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.