Saturday 11th September 1976. That was the day that my Dad first took me to see the (occasionally) mighty West Ham United.
The opponents, the rather more often mighty Arsenal.
I still have vivid memories of that game. The noise from the crowd. The fact that the grass looked so green, brighter than on the TV.
West Ham not playing very well. Frank Stapleton putting a bit of a downer on the day by having the temerity to score twice in a 2-0 win for the Gunners.
My Dad recently celebrated his 70th birthday.
His present from his first-born son ? A trip to see the (previously) mighty Luton Town take on the ( probably must have been from time to time) mighty Nuneaton Borough.
Now, this may seem poor reward for my dear old Dad – he takes me to see two of the top teams in the country and he gets the Blue Square Premier League in return.
Additionally, these days it’s less the colour of the grass that assaults the senses than the colour of the boots.
These are various flourescent colours, virtually none of them black.
Mind you, as Deb pointed out, if you play for Luton and spend most of your working life dressed in bright orange, then accessorising must be a bit of a challenge.
The game itself however, is another matter.
Typical English Football – very quick, lots of commitment. You can tell it’s not the Premiership by the absence of millionaires rolling around the floor in apparent agony because they’ve broken a finger-nail.
Two late goals sends the Hatters home happy.
All of which has nothing to do with the subject of this post, apart from my choice of examples.
Comparing the table structure between different schemas is standard functionality for any self-respecting IDE. However, things get a bit more tricky if you’ve had a visit from the Prefix Pixie. He, she (or it if you’re table relationship diagram dropped out of a design tool) thought it’d be a good idea to give the same prefix to every table in the schema.
The result of this is that the tools in the IDE can’t recognize that tables with different names are meant to have identical structures.
So much for the “Premiership” of Database Development, it looks like we’ll just have to do a bit of D.I.Y. to see through the poxie pixie dust.
Dad would approve.
The tables
In this example, we have a table to hold details of Luton’s current meteoric…er…not-quite-rise-to-the-top of the league.
We also have a separate schema to hold archived records from the current table. The current table looks like this :
CREATE TABLE cmonu_hatters( season NUMBER(4), pld NUMBER(2), won NUMBER(2), drawn NUMBER(2), lost NUMBER(2), gl_for NUMBER(2), gl_ag NUMBER(2)) / INSERT INTO cmonu_hatters(season, pld, won, drawn, lost, gl_for, gl_ag) VALUES( 2009, 44, 26, 10, 8, 84, 40) / INSERT INTO cmonu_hatters(season, pld, won, drawn, lost, gl_for, gl_ag) VALUES( 2010, 46, 23, 15, 8, 85, 37) / INSERT INTO cmonu_hatters(season, pld, won, drawn, lost, gl_for, gl_ag) VALUES( 2011, 46, 22, 15, 9, 78, 42) / COMMIT;
As for the archive table, that’s in a separate schema :
CREATE user hatter identified by pwd; ALTER USER hatter default tablespace users QUOTA UNLIMITED ON users; -- Now create the table CREATE TABLE hatter.old_hatters( season NUMBER(4), pld NUMBER(2), won NUMBER(2), drawn NUMBER(2), lost NUMBER(2), gl_for NUMBER(2), gl_ag NUMBER(2)) /
Keeping these tables in synch is going to be an ongoing maintenance overhead.
Say, for example, we want to make some changes to the master table :
ALTER TABLE cmonu_hatters MODIFY ( season NOT NULL); -- Simon's got a good feeling about this season, so increase the size of gl_for ALTER TABLE cmonu_hatters MODIFY( gl_for NUMBER(3));
Immediately we can see there might be problems. the OLD_HATTERS table only has gl_for defined as a NUMBER(2).
If Simon is right, then when we come to insert this season’s results into the table there will be problems.
There’s a Dictionary View for that
As I’ve observed before, there is a wealth of information in the Data Dictionary that people tend to forget about.
In this case, it’s DBA_TAB_COLS that can come to our rescue :
SELECT tgt.table_name, tgt.column_name, src.data_type,
src.data_length, src.data_precision, src.data_scale,
src.nullable
FROM sys.dba_tab_cols src, sys.dba_tab_cols tgt
WHERE REPLACE(src.table_name, 'CMONU') = REPLACE(tgt.table_name, 'OLD')
AND src.column_name = tgt.column_name
AND src.owner = 'MIKE'
AND tgt.owner = 'HATTER'
AND(
tgt.data_type != src.data_type
OR tgt.data_length != src.data_length
OR tgt.nullable != src.nullable
OR tgt.data_precision != src.data_precision
OR tgt.data_scale != src.data_scale);
When we run this we can see the differences :
SQL> / TABLE_NAME COLUMN_NAME DATA_TYPE DATA_LENGTH DATA_PRECISION DATA_SCALE N -------------------- -------------------- --------------- ----------- -------------- ---------- - OLD_HATTERS GL_FOR NUMBER 22 3 0 Y OLD_HATTERS SEASON NUMBER 22 4 0 N SQL>
We can also use this view to identify any columns that exist in the master table but not in the target :
ALTER TABLE cmonu_hatters ADD( position NUMBER(2)); UPDATE cmonu_hatters SET position = 2 WHERE season = 2009 / UPDATE cmonu_hatters SET position = 3 WHERE season = 2010 / UPDATE cmonu_hatters SET position = 5 WHERE season = 2011 / COMMIT;
To find this change :
SELECT DISTINCT tgt.table_name, src.column_name, src.data_type,
src.data_length, src.data_precision, src.data_scale,
src.nullable
FROM sys.dba_tab_cols src, sys.dba_tab_cols tgt
WHERE REPLACE(src.table_name, 'CMONU_') = REPLACE( tgt.table_name, 'OLD_')
AND src.owner = 'MIKE'
AND tgt.owner = 'HATTER'
AND NOT EXISTS(
SELECT 1
FROM sys.dba_tab_cols tgt1
WHERE tgt1.table_name = tgt.table_name
AND tgt1.owner = tgt.owner
AND tgt1.column_name = src.column_name);
Run this and we get :
TABLE_NAME COLUMN_NAME DATA_TYPE DATA_LENGTH DATA_PRECISION DATA_SCALE N -------------------- -------------------- --------------- ----------- -------------- ---------- - OLD_HATTERS POSITION NUMBER 22 2 0 Y SQL>
Generating DDL…carefully
At this point, it would be a simple matter to generate the required DDL to synchronize the archive table with the master table. However, as always, we need to be a bit careful here. Remember, there is more than one way to inject SQL into a database.
The statements we want to generate follow the format :
ALTER TABLE schema.table name MODIFY column name datatype
… for column changes and :
ALTER TABLE schema.table name ADD column name datatype
… for new columns.
To build this safely, we need to be sure that the schema, table_name and column_name do not contain any nasty surprises of the SQL Injection variety.
At this point, knowing our application, we can reasonably assume that all of these objects follow these naming conventions :
- they contain only alphanumeric characters and the underscore character
- they do not contain spaces
We also know that, both the schemas and table_names are already valid object names in the database as we’re retrieving them from the data dictionary. Therfore utilities such as DBMS_ASSERT.SCHEMA_NAME are not going to tell us anything we don’t already know.
What we really need to do is to check each of the items we’re going to concatenate into our DDL to make sure that there’s nothing untoward happening. To do this, we’ll just knock together a simple function like this :
CREATE OR REPLACE FUNCTION is_clean_fn( i_object_name IN VARCHAR2)
RETURN BOOLEAN IS
-------------------------------------------------------------------------------
-- Function to determine whether input that may be used to dynamically
-- build statements contains "allowable" characters or whether it may
-- possibly be an injectable statement.
-- The assumption is that an object name ( in this case, including user supplied
-- parameter values), will contain only alphanumeric and '_' characters.
--
-------------------------------------------------------------------------------
BEGIN
--
-- Make sure that this object name does not contain any spaces or
-- punctuation, apart from '_'
--
IF REGEXP_INSTR( REPLACE( i_object_name, '_'), '[[:punct:]]|[[:space:]]') > 0
THEN
RETURN FALSE;
END IF;
RETURN TRUE;
END is_clean_fn;
/
Now just to give this a quick test…
set serveroutput on size unlimited
DECLARE
--
-- Test of is_clean_fn function
--
TYPE typ_test_strings IS TABLE OF VARCHAR2(30) INDEX BY PLS_INTEGER;
tbl_test_strings typ_test_strings;
l_test_schema VARCHAR2(30) := 'MIKE';
l_test_col VARCHAR2(30) := 'GLS_FOR';
l_test_tab VARCHAR2(30) := 'CMONU_HATTERS';
l_test_bad VARCHAR2(30) := 'OR 1=1 --';
BEGIN
tbl_test_strings(1) := 'MIKE'; -- a schema name
tbl_test_strings(2) := 'GLS_FOR'; -- a column_name
tbl_test_strings(3) := 'CMONU_HATTERS'; -- a table name
tbl_test_strings(4) := ' OR 1=1 --'; -- some kind of SQL Injection attempt
FOR i IN 1..tbl_test_strings.COUNT LOOP
IF is_clean_fn( i_object_name => tbl_test_strings(i)) THEN
DBMS_OUTPUT.PUT_LINE( tbl_test_strings(i)||' is OK.');
ELSE
DBMS_OUTPUT.PUT_LINE(tbl_test_strings(i)||' looks a bit suspicious to me.');
END IF;
END LOOP;
END;
/
… and the results …
SQL> @clean_test.sql MIKE is OK. GLS_FOR is OK. CMONU_HATTERS is OK. OR 1=1 -- looks a bit suspicious to me. PL/SQL procedure successfully completed. SQL>
Right, now to put it all together in a neat package…
CREATE OR REPLACE PACKAGE synch_tabs_pkg AS
FUNCTION is_clean_fn( i_object_name IN VARCHAR2) RETURN BOOLEAN;
FUNCTION get_missing_cols_fn(
i_source_schema IN all_users.username%TYPE,
i_source_fix IN VARCHAR2,
i_target_schema IN all_users.username%TYPE,
i_target_fix IN VARCHAR2)
RETURN CLOB;
FUNCTION get_col_diffs_fn(
i_source_schema IN all_users.username%TYPE,
i_source_fix IN VARCHAR2,
i_target_schema IN all_users.username%TYPE,
i_target_fix IN VARCHAR2)
RETURN CLOB;
FUNCTION get_schema_diffs_fn(
i_source_schema IN all_users.username%TYPE,
i_source_fix IN VARCHAR2,
i_target_schema IN all_users.username%TYPE,
i_target_fix IN VARCHAR2)
RETURN CLOB;
TYPE rec_cols IS RECORD(
table_name dba_tab_cols.table_name%TYPE,
column_name dba_tab_cols.column_name%TYPE,
data_type dba_tab_cols.data_type%TYPE,
data_length dba_tab_cols.data_length%TYPE,
data_precision dba_tab_cols.data_precision%TYPE,
data_scale dba_tab_cols.data_scale%TYPE,
nullable dba_tab_cols.nullable%TYPE);
TYPE typ_cols IS TABLE OF rec_cols INDEX BY PLS_INTEGER;
END synch_tabs_pkg;
/
CREATE OR REPLACE PACKAGE BODY synch_tabs_pkg AS
FUNCTION is_clean_fn( i_object_name IN VARCHAR2)
RETURN BOOLEAN IS
-------------------------------------------------------------------------------
-- Function to determine whether input that may be used to dynamically
-- build statements contains "allowable" characters or whether it may
-- possibly be an injectable statement.
-- The assumption is that an object name ( in this case, including user supplied
-- parameter values), will contain only alphanumeric and '_' characters.
--
-------------------------------------------------------------------------------
BEGIN
--
-- Make sure that this object name does not contain any spaces or
-- punctuation, apart from '_'
--
IF REGEXP_INSTR( REPLACE( i_object_name, '_'), '[[:punct:]]|[[:space:]]') > 0
THEN
RETURN FALSE;
END IF;
RETURN TRUE;
END is_clean_fn;
FUNCTION get_missing_cols_fn(
i_source_schema IN all_users.username%TYPE,
i_source_fix IN VARCHAR2,
i_target_schema IN all_users.username%TYPE,
i_target_fix IN VARCHAR2)
RETURN CLOB
IS
-------------------------------------------------------------------------------
-- Function to find any columns that exist in the source schema tables but not
-- in the target. For any that are found, build the appropriate DDL statement.
-- Return all the required statements as a single CLOB.
--
-------------------------------------------------------------------------------
tbl_new_cols typ_cols;
l_err_msg VARCHAR2(4000);
l_ddl VARCHAR2(4000);
l_rtn CLOB := EMPTY_CLOB();
e_not_clean EXCEPTION;
BEGIN
--
-- Check that the source and target schemas contain "valid" characters
--
IF NOT is_clean_fn( i_object_name => i_source_schema) THEN
l_err_msg := i_source_schema||' contains invalid characters. Please investigate';
RAISE e_not_clean;
END IF;
IF NOT is_clean_fn( i_object_name => i_target_schema) THEN
l_err_msg := i_target_schema||' contains invalid characters. Please investigate';
RAISE e_not_clean;
END IF;
--
-- Find columns in the source schema tables that do not exist in the
-- corresponding target schema tables.
--
SELECT DISTINCT tgt.table_name, src.column_name, src.data_type,
src.data_length, src.data_precision, src.data_scale,
src.nullable
BULK COLLECT INTO tbl_new_cols
FROM sys.dba_tab_cols src, sys.dba_tab_cols tgt
WHERE REPLACE(src.table_name, i_source_fix) = REPLACE( tgt.table_name, i_target_fix)
AND src.owner = i_source_schema
AND tgt.owner = i_target_schema
AND NOT EXISTS(
SELECT 1
FROM sys.dba_tab_cols tgt1
WHERE tgt1.table_name = tgt.table_name
AND tgt1.owner = tgt.owner
AND tgt1.column_name = src.column_name);
--
-- Check to see if we've found any columns...
--
IF tbl_new_cols.COUNT = 0 THEN
l_rtn := '-- No columns to add.';
RETURN l_rtn;
END IF;
--
-- If we're still here then we have found columns to add. We now need
-- to build the appropriate DDL statements to add these columns.
-- To do this :
-- - make sure that the values we'll be using are not likely to be
-- part of a SQL Injection
-- - Build the appropriate statement based on the datatype of the new column.
-- - If the new column is not null then include a not null constraint in the
-- DDL statement.
-- NOTE - this will work for columns of type NUMBER, DATE and VARCHAR2 ONLY.
--
-- The statement formats are as follows :
-- - for Numeric datatypes
-- ALTER TABLE owner.table_name ADD column_name(precision, scale)
-- - for VARCHAR :
-- ALTER TABLE owner.table_name ADD column_name(length)
-- - for DATE
-- ALTER TABLE owner.table_name ADD column_name
--
FOR i IN 1..tbl_new_cols.COUNT LOOP
IF NOT is_clean_fn( i_object_name => tbl_new_cols(i).table_name)
OR NOT is_clean_fn( i_object_name => tbl_new_cols(i).column_name)
THEN
l_err_msg := tbl_new_cols(i).table_name
||'.'||tbl_new_cols(i).column_name
||' contains invalid characters. Please investigate.';
RAISE e_not_clean;
END IF;
--
-- Provided the table and column names are OK, build the DDL...
--
l_ddl := 'ALTER TABLE '||i_target_schema||'.'||tbl_new_cols(i).table_name
||' ADD ('||tbl_new_cols(i).column_name
||' '||tbl_new_cols(i).data_type;
IF tbl_new_cols(i).data_type = 'VARCHAR2' THEN
l_ddl := l_ddl||'('||tbl_new_cols(i).data_length||')';
ELSIF tbl_new_cols(i).data_precision IS NOT NULL THEN
--
-- numeric datatype so include the data_scale...
--
l_ddl := l_ddl||'('||tbl_new_cols(i).data_precision||','
||tbl_new_cols(i).data_scale||')';
END IF;
IF tbl_new_cols(i).nullable = 'N' THEN
--
-- Add a NOT NULL constraint
--
l_ddl := l_ddl||' NOT NULL';
END IF;
l_ddl := l_ddl || ');';
l_rtn := l_rtn||CHR(10)||l_ddl;
END LOOP;
RETURN l_rtn;
EXCEPTION
WHEN e_not_clean THEN
RAISE_APPLICATION_ERROR(-20000, l_err_msg);
END get_missing_cols_fn;
FUNCTION get_col_diffs_fn(
i_source_schema IN all_users.username%TYPE,
i_source_fix IN VARCHAR2,
i_target_schema IN all_users.username%TYPE,
i_target_fix IN VARCHAR2)
RETURN CLOB
IS
-------------------------------------------------------------------------------
-- Function to find any differences in column definitions between the source
-- and target schemas.
-- For any that are found, build the appropriate DDL statement.
-- Return all the required statements as a single CLOB.
--
-------------------------------------------------------------------------------
tbl_diff_cols typ_cols;
l_ddl VARCHAR2(4000);
l_rtn CLOB := EMPTY_CLOB();
l_err_msg VARCHAR2(4000);
e_not_clean EXCEPTION;
BEGIN
--
-- Check that the source and target schemas contain "valid" characters
--
IF NOT is_clean_fn( i_object_name => i_source_schema) THEN
l_err_msg := i_source_schema||' contains invalid characters. Please investigate';
RAISE e_not_clean;
END IF;
IF NOT is_clean_fn( i_object_name => i_target_schema) THEN
l_err_msg := i_target_schema||' contains invalid characters. Please investigate';
RAISE e_not_clean;
END IF;
SELECT DISTINCT tgt.table_name, tgt.column_name, src.data_type,
src.data_length, src.data_precision, src.data_scale,
src.nullable
BULK COLLECT INTO tbl_diff_cols
FROM sys.dba_tab_cols src, sys.dba_tab_cols tgt
WHERE REPLACE(src.table_name, i_source_fix) = REPLACE(tgt.table_name, i_target_fix)
AND src.column_name = tgt.column_name
AND src.owner = i_source_schema
AND tgt.owner = i_target_schema
AND(
tgt.data_type != src.data_type
OR tgt.data_length != src.data_length
OR tgt.nullable != src.nullable
OR tgt.data_precision != src.data_precision
OR tgt.data_scale != src.data_scale);
IF tbl_diff_cols.COUNT = 0 THEN
-- No differences
l_ddl := '-- No columns to amend.';
RETURN l_ddl;
END IF;
--
-- If we're still here then some columns need to be amended...
--
FOR i IN 1..tbl_diff_cols.COUNT LOOP
--
-- Check that the table and column names are OK...
--
IF NOT is_clean_fn( i_object_name => tbl_diff_cols(i).table_name)
OR NOT is_clean_fn( i_object_name => tbl_diff_cols(i).column_name)
THEN
l_err_msg := tbl_diff_cols(i).table_name||'.'||tbl_diff_cols(i).column_name
||' contains invalid characters. Please investigate.';
RAISE e_not_clean;
END IF;
l_ddl := 'ALTER TABLE '||i_target_schema||'.'||tbl_diff_cols(i).table_name
||' MODIFY ( '||tbl_diff_cols(i).column_name;
--
-- The rest of the statement we're building is dependent on the data type
--
IF tbl_diff_cols(i).data_precision IS NOT NULL THEN
--
-- This column has a numeric datatype so include the precision and scale.
-- According to Oracle Docs on DBA_TAB_COLUMNS...
-- DATA_PRECISION - Decimal precision for NUMBER datatype;
-- binary precision for FLOAT datatype; NULL for all other datatypes
--
-- NOTE if a column is just defined as NUMBER with no precision,
-- no problem, we won't get here.
--
l_ddl := l_ddl||' '||tbl_diff_cols(i).data_type||'('
||tbl_diff_cols(i).data_precision;
IF tbl_diff_cols(i).data_scale IS NOT NULL THEN
l_ddl := l_ddl||','||tbl_diff_cols(i).data_scale;
END IF;
l_ddl := l_ddl||') ';
ELSIF tbl_diff_cols(i).data_type = 'DATE' THEN
--
-- Just the datatype required.
--
l_ddl := l_ddl||' '||tbl_diff_cols(i).data_type||' ';
ELSE
--
-- Most commonly a VARCHAR2 - need the data_length
--
l_ddl := l_ddl||' '||tbl_diff_cols(i).data_type||'( '||tbl_diff_cols(i).data_length||') ';
END IF;
--
-- Finally, add the NOT NULL constraint if required...
--
IF tbl_diff_cols(i).nullable = 'N' THEN
l_ddl := l_ddl||' NOT NULL';
END IF;
l_ddl := l_ddl ||');';
l_rtn := l_rtn||CHR(10)||l_ddl;
END LOOP;
RETURN l_rtn;
EXCEPTION
WHEN e_not_clean THEN
RAISE_APPLICATION_ERROR(-20001, l_err_msg);
END get_col_diffs_fn;
FUNCTION get_schema_diffs_fn(
i_source_schema IN all_users.username%TYPE,
i_source_fix IN VARCHAR2,
i_target_schema IN all_users.username%TYPE,
i_target_fix IN VARCHAR2)
RETURN CLOB
IS
-------------------------------------------------------------------------------
-- Main entry point to the package.
-- This function checks the input parameters.
-- It then calls the get_missing_cols_fn function for DDL to add missing
-- columns to the target schema.
-- It calls get_cols_diffs_fn for DDL to amend columns in the target schema.
-- Finally, it outputs the DDL in the form of a CLOB which can then be
-- executed as a file...after being checked by a Human !
--
-------------------------------------------------------------------------------
l_err_msg VARCHAR2(4000);
l_missing CLOB := EMPTY_CLOB();
l_diffs CLOB := EMPTY_CLOB();
l_rtn CLOB := EMPTY_CLOB();
e_missing_schema EXCEPTION;
e_not_clean EXCEPTION;
BEGIN
--
-- Check the input parameters. The only mandatory parameters are the
-- source and target schema names...
--
IF i_source_schema IS NULL OR i_target_schema IS NULL THEN
l_err_msg := 'Need to provide a source and a target schema.';
RAISE e_missing_schema;
END IF;
--
-- Make sure that the schemas do not contain "dodgy" characters...
-- NOTE - the prefix/suffix strings are not used in the generated DDL statements
-- directly so pose no threat from an injection point of view.
--
IF NOT is_clean_fn( i_object_name => i_source_schema) THEN
l_err_msg := i_source_schema||' contains invalid characters. Please investigate';
RAISE e_not_clean;
END IF;
IF NOT is_clean_fn( i_object_name => i_target_schema) THEN
l_err_msg := i_target_schema||' contains invalid characters. Please investigate';
RAISE e_not_clean;
END IF;
--
-- Now find any missing columns...
--
l_missing := get_missing_cols_fn(
i_source_schema => i_source_schema,
i_source_fix => i_source_fix,
i_target_schema => i_target_schema,
i_target_fix => i_target_fix);
--
-- ...and any column definition changes...
--
l_diffs := get_col_diffs_fn(
i_source_schema => i_source_schema,
i_source_fix => i_source_fix,
i_target_schema => i_target_schema,
i_target_fix => i_target_fix);
--
-- Finally, concatenate the results from the two calls into one
-- CLOB to return...
--
l_rtn := '-- Columns to Add : '||CHR(10)||l_missing||CHR(10)
||'-- Columns to Modify : '||CHR(10)||l_diffs;
RETURN l_rtn;
EXCEPTION
WHEN e_missing_schema THEN
RAISE_APPLICATION_ERROR(-20003, l_err_msg);
WHEN e_not_clean THEN
RAISE_APPLICATION_ERROR( -20004, l_err_msg);
END get_schema_diffs_fn;
END synch_tabs_pkg;
/
And now run this…
set heading off
set lines 4000
set pages 5000
SELECT synch_tabs_pkg.get_schema_diffs_fn(
i_source_schema => 'MIKE',
i_source_fix => 'CMONU',
i_target_schema => 'HATTER',
i_target_fix => 'OLD')
FROM dual
/
What we end up with is output that looks like this :
-- Columns to Add : ALTER TABLE HATTER.OLD_HATTERS ADD (POSITION NUMBER(2,0)); -- Columns to Modify : ALTER TABLE HATTER.OLD_HATTERS MODIFY ( GL_FOR NUMBER(3,0) ); ALTER TABLE HATTER.OLD_HATTERS MODIFY ( SEASON NUMBER(4,0) NOT NULL);
I’ve taken the precaution of outputting these statements rather than executing them in the code so that I can double-check them rather than risk something unfortunate happening to my tables.
Funny thing nostalgia…I remember back when England batsmen couldn’t play spin and their spinners were absolutely awful at bowling out Indian batsmen. Fortunately, I would like to think that those days were called the 20th century (although I fear they could also be called…next week).
Filed under: Oracle, PL/SQL, SQL Tagged: ALTER TABLE ADD, ALTER TABLE MODIFY, DBA_TAB_COLS, REGEXP_INSTR, REPLACE, SQL Injection
