Prescient Analytics - Blog #SQL

Handling "Valid" SQL Join Duplication

Fri, 06 Dec 2019 12:45:10 +1030

A hump that most people starting out in SQL have to get over is how joins work - that is, thinking relational.

A scenario that came up recently involved matching payments in two different systems, the purpose of which was to trace it back to a source vendor.

An issue I experienced was the case of “valid” duplicate joining – that is there were two transactions in one table and two in the other and both had the same join column values. Therefore the result with relational joining is that four records are returned since they all join to each other, rather than the desired two.

One way to remove these duplicates is to use good old row_number() to generate a pseudo-key to filter them out.

This is somewhat arbitrary depending on the data, since the only control you have is in the order by clause of the row_number() function. In my use case this was acceptable since other key conditions were met in the join clause.

Below is an example demonstrating this approach:

create temporary table test_legit_matches_1 (

idx1 int,

col1 text,

join_col text

);

create temporary table test_legit_matches_2 (

idx2 int,

col2 text,

join_col text

);

insert into test_legit_matches_1 values (1, 'val1', 'join me');

insert into test_legit_matches_1 values (2, 'val2', 'join me');

insert into test_legit_matches_1 values (3, 'val3', 'nice join');

insert into test_legit_matches_2 values (1, 'val4', 'join me');

insert into test_legit_matches_2 values (2, 'val5', 'join me');

insert into test_legit_matches_2 values (3, 'val6', 'nice join');

-- observed duplication

select

from test_legit_matches_1 as t1

left outer join test_legit_matches_2 as t2

on t1.join_col = t2.join_col

;

-- duplication removed, though join is potentially arbitrary (based on order by clauses)

with _joined as (

select

, row_number() over (partition by idx1 order by idx2) as idx1_rn

, row_number() over (partition by idx2 order by idx1) as idx2_rn

from test_legit_matches_1 as t1

left outer join test_legit_matches_2 as t2

on t1.join_col = t2.join_col

)

select *

from _joined

where idx1_rn = idx2_rn

;

Result 1:

Result 2:

Grouping Data by Attribute Using Lag & Cumulative Sum

Thu, 21 Feb 2019 15:11:54 +1030

A fair proportion of my work involves aspects of data quality i.e. munging data.

Routinely, there is a requirement to identify duplicate groups of records.

The following is a snippet that creates group ids from which further processing can be performed

1. Setup an example table

create table #tmp_cust (
  id int,
  group_field varchar(max),
);

insert into #tmp_cust values (1, 'attr1');
insert into #tmp_cust values (2, 'attr1');
insert into #tmp_cust values (3, 'attr1');
insert into #tmp_cust values (4, 'attr2');
insert into #tmp_cust values (5, 'attr2');
insert into #tmp_cust values (6, 'attr3');

2. Use the lag function to determine when a the group field changes

with lagged as (
    select
    id
    , group_field
    , case when group_field <> lag(group_field) over (partition by 1 order by group_field, id) then 1 else 0 end as Group_Flag
  from #tmp_cust tc
)

3. Use a cumulative sum; achieved by rows unbounded preceding in the window function.

select
  id
  , group_field
  , sum(Group_Flag) over (order by group_field, id rows unbounded preceding ) as Group_Flag
from lagged

Results:

With the groups identified we can then move onto removing/remediation of duplicates!

(Tested using SQL Server 2016)

Grouping Temporal Sequences in SQL

Mon, 08 Oct 2018 22:45:00 +1030

I was recently working in a PostgreSQL (9.6) environment and had one of those problems that comes around infrequently enough that it isn't a worn pattern in my brain. Contributing to this is the fact that I dwelt for a long time in SQL Server 2008 land which lacked a lot of functionality (lead/lag functions) that have been in the wild for quite a while.

Problem: Temporal transaction data that needed to be grouped into sequences. In this scenario, there is also an additional column by which the data is to be grouped.

The set-up code to demonstrate the approach:


drop table if exists seq_demo
;
create table seq_demo (
    dt date
    , grouping character varying
);
insert into seq_demo values ('2018-01-01', 'grp1');
insert into seq_demo values ('2018-01-02', 'grp1');
insert into seq_demo values ('2018-01-04', 'grp1');
insert into seq_demo values ('2018-01-05', 'grp1');
insert into seq_demo values ('2018-01-06', 'grp1');
insert into seq_demo values ('2018-01-03', 'grp2');
insert into seq_demo values ('2018-01-07', 'grp2');
insert into seq_demo values ('2018-01-08', 'grp2');
insert into seq_demo values ('2018-01-09', 'grp2');
insert into seq_demo values ('2018-01-10', 'grp2');

The above test data creates four sequences across the two groupings: { 'grp1', 'grp2' }

And we would like to transform this into the following output:

1	grp1	2018-01-01	2018-01-02
2	grp1	2018-01-04	2018-01-06
3	grp2	2018-01-03	2018-01-03
4	grp2	2018-01-07	2018-01-10

This is one approach that makes use of the lag function, following by a cumulative (windowed) sum to create the groups.

with lagged as (
    select
        dt
        , grouping
        , lag(dt) over (partition by grouping order by dt) as lagged_dt
        -- the rn preserves row order for the next step, which is sometimes desired
        , row_number() over (partition by grouping order by dt) as rn
    from seq_demo
), grouped as (
    select
        dt
        , grouping
        , lagged_dt
        , (dt - lagged_dt) as days_diff
        , case when lagged_dt is null or (dt - lagged_dt) > 1 then 1 else 0 end as changed_dt
        , sum(case when lagged_dt is null or (dt - lagged_dt) > 1 then 1 else 0 end)
           over (partition by grouping order by rn rows between unbounded preceding and current row) as dt_group
        -- note on postgresql "rows between unbounded preceding and current row" was not required

    from lagged
)
select
    grouping
    , dt_group
    , min(dt) as start_dt
    , max(dt) as end_dt
from grouped

group by
    grouping
    , dt_group

order by
    grouping
    , dt_group
;

My results:

The specific use case this solved was to identify isolated transactions that had a group count of 1.

This pattern also works well to condense repeated dimensional data that appears in a transactional form.