Prescient Analytics - Blog by kpauligk

Handling "Valid" SQL Join Duplication

Fri, 06 Dec 2019 12:45:10 +1030

A hump that most people starting out in SQL have to get over is how joins work - that is, thinking relational.

A scenario that came up recently involved matching payments in two different systems, the purpose of which was to trace it back to a source vendor.

An issue I experienced was the case of “valid” duplicate joining – that is there were two transactions in one table and two in the other and both had the same join column values. Therefore the result with relational joining is that four records are returned since they all join to each other, rather than the desired two.

One way to remove these duplicates is to use good old row_number() to generate a pseudo-key to filter them out.

This is somewhat arbitrary depending on the data, since the only control you have is in the order by clause of the row_number() function. In my use case this was acceptable since other key conditions were met in the join clause.

Below is an example demonstrating this approach:

create temporary table test_legit_matches_1 (

idx1 int,

col1 text,

join_col text

);

create temporary table test_legit_matches_2 (

idx2 int,

col2 text,

join_col text

);

insert into test_legit_matches_1 values (1, 'val1', 'join me');

insert into test_legit_matches_1 values (2, 'val2', 'join me');

insert into test_legit_matches_1 values (3, 'val3', 'nice join');

insert into test_legit_matches_2 values (1, 'val4', 'join me');

insert into test_legit_matches_2 values (2, 'val5', 'join me');

insert into test_legit_matches_2 values (3, 'val6', 'nice join');

-- observed duplication

select

from test_legit_matches_1 as t1

left outer join test_legit_matches_2 as t2

on t1.join_col = t2.join_col

;

-- duplication removed, though join is potentially arbitrary (based on order by clauses)

with _joined as (

select

, row_number() over (partition by idx1 order by idx2) as idx1_rn

, row_number() over (partition by idx2 order by idx1) as idx2_rn

from test_legit_matches_1 as t1

left outer join test_legit_matches_2 as t2

on t1.join_col = t2.join_col

)

select *

from _joined

where idx1_rn = idx2_rn

;

Result 1:

Result 2:

The type initializer for 'Amazon.Runtime.Internal.Settings.PersistenceManager' threw an exception

Thu, 01 Aug 2019 22:31:02 +0930

I was faced with this lovely error message from a process that had been running successfully for some time.

It was a Powershell script running in an Azure Function App, and my initial internet research did not reveal many clues as to why this would be case.

On a hunch, I thought to update the AWS Powershell modules it was using - though they were not particularly old, I figured it couldn't hurt. So I did exactly that and voila, I tested the process and we were back in business.... for a few days least.

The exception was back.

I stumbled on a clue whilst reading a stack overflow response pointing at the parameters of an AWS function the OP was having trouble with. Following this, I double checked the parameters of the function that was failing (Get-S3Object) and found I hadn't set the Region parameter. I added the parameter, tested it and no exception! This looks to have been the root cause, though why it worked up to that point is a bit of a mystery to me. As an aside, I have not had this issue ever in my local environment (in Powershell ISE).

Perhaps it should be made a required parameter given the potential exception it may raise though.

I hope this helps someone, because it was pretty obscure...to me at least!

Data Analytics Project Blueprint

Thu, 02 May 2019 10:18:57 +0930

We recently had the opportunity to put some of our experience to paper in order to assist a client who is seeking to implement organisational analytics in a meaningful way for the first time.

The attached image is the high level 'blueprint' put together that we believe encompasses the main activities that need to be undertaken to be ultimately successful in execution. Naturally behind each is a lot more detail on specifics of the activity, the outputs generated and the relationship with other activities.

Important to note though is that this does not override project management activities, merely complements it with specific items that relate to data and analytics projects. I.E. it specifically does not refer to activities like risk management, budget management, communications plans etc.

First Look at OR-Tools

Thu, 02 May 2019 10:18:04 +0930

For a recent project that involved an optimisation problem, I settled on using the OR-Tools library, an open-source project from Google (https://developers.google.com/optimization).

Of interest to me was its CP-SAT solver which claimed good solution speed based on Constraint Programming (https://en.wikipedia.org/wiki/Constraint_programming)

Key selling points:

Documentation looked good with plenty of examples – this was critical since I was starting from zero and needed to get up to speed quickly;
Has a Python interface, which is the only programming language I consider myself a capable developer in;
Supports a number of solver back-ends; although I did not get a chance to try this out; and
If Google uses it, it must be good... right?

As someone who has more experience with Stats/Machine Learning models, Constraint Optimisation presented an interesting challenge given the way models need to be defined;

Variables must be integers, this means multiplying decimal variables to a magnitude that allows the truncation of the decimal component.
Expression are in the form of linear equations, which creates some limitations. For example you are unable to simply multiply variables within the same expression such as y = a * b, however you can work around this; in this case one has to use a function cp_model.AddProdEquality(, [])

To note however, that the CP-SAT solver although superior, has a different Python API to the previous CP Solver which (on face value) appeared to have more flexibility in how equations could be constructed.

Also, more generally, when referring to the API docs (which are presented in C++) there appears to be certain things that are either not represented in the Python API, or not well documented to understand how it should be used.

I’ll dig into things a bit further in future posts as I’d like pursue some experimentation to understand pros and cons of different approaches. In the meantime check out the examples, they really are quite comprehensive for standard problems.

As a postscript, the project was successfully delivered using OR-Tools , however due to the modeled variables and constraints, solution speed was pretty slow. This was not unexpected and not a reflection of OR-Tools since the model contained thousands of variables … but it would have been nice to find a silver bullet!

Grouping Data by Attribute Using Lag & Cumulative Sum

Thu, 21 Feb 2019 15:11:54 +1030

A fair proportion of my work involves aspects of data quality i.e. munging data.

Routinely, there is a requirement to identify duplicate groups of records.

The following is a snippet that creates group ids from which further processing can be performed

1. Setup an example table

create table #tmp_cust (
  id int,
  group_field varchar(max),
);

insert into #tmp_cust values (1, 'attr1');
insert into #tmp_cust values (2, 'attr1');
insert into #tmp_cust values (3, 'attr1');
insert into #tmp_cust values (4, 'attr2');
insert into #tmp_cust values (5, 'attr2');
insert into #tmp_cust values (6, 'attr3');

2. Use the lag function to determine when a the group field changes

with lagged as (
    select
    id
    , group_field
    , case when group_field <> lag(group_field) over (partition by 1 order by group_field, id) then 1 else 0 end as Group_Flag
  from #tmp_cust tc
)

3. Use a cumulative sum; achieved by rows unbounded preceding in the window function.

select
  id
  , group_field
  , sum(Group_Flag) over (order by group_field, id rows unbounded preceding ) as Group_Flag
from lagged

Results:

With the groups identified we can then move onto removing/remediation of duplicates!

(Tested using SQL Server 2016)

GnuGPG - How to fix gpg: handle plaintext failed: General error

Wed, 23 Jan 2019 08:34:47 +1030

[Note: this issue occured using GPG4Win 2.2.5 in an Azure Function App environment]

I recently encountered a frustrating, albeit minor, issue. A Powershell Script process that was running on as an Azure Function App, started throwing an error gpg: handle plaintext failed: General error.

A quick Google search of the error message did not reveal much, though the first result did refer to filenames potentially being a cause.

The frustrating part was that it was working correctly previously, and on my local environment there was no issue in the command I was using: bin\gpg.exe -v --batch --output d:\home\download\files\file_07112018.zip --decrypt D:\home\download\files\file_07112018.zip.gpg

Long story (of trying error logging, different switches etc) short, I altered the output filename and placed an underscore suffix and magically it worked! A guess is a conflict between the filename set in the encrypted file since it was the same as I was trying to output.

Hope this helps someone!

Grouping Temporal Sequences in SQL

Mon, 08 Oct 2018 22:45:00 +1030

I was recently working in a PostgreSQL (9.6) environment and had one of those problems that comes around infrequently enough that it isn't a worn pattern in my brain. Contributing to this is the fact that I dwelt for a long time in SQL Server 2008 land which lacked a lot of functionality (lead/lag functions) that have been in the wild for quite a while.

Problem: Temporal transaction data that needed to be grouped into sequences. In this scenario, there is also an additional column by which the data is to be grouped.

The set-up code to demonstrate the approach:


drop table if exists seq_demo
;
create table seq_demo (
    dt date
    , grouping character varying
);
insert into seq_demo values ('2018-01-01', 'grp1');
insert into seq_demo values ('2018-01-02', 'grp1');
insert into seq_demo values ('2018-01-04', 'grp1');
insert into seq_demo values ('2018-01-05', 'grp1');
insert into seq_demo values ('2018-01-06', 'grp1');
insert into seq_demo values ('2018-01-03', 'grp2');
insert into seq_demo values ('2018-01-07', 'grp2');
insert into seq_demo values ('2018-01-08', 'grp2');
insert into seq_demo values ('2018-01-09', 'grp2');
insert into seq_demo values ('2018-01-10', 'grp2');

The above test data creates four sequences across the two groupings: { 'grp1', 'grp2' }

And we would like to transform this into the following output:

1	grp1	2018-01-01	2018-01-02
2	grp1	2018-01-04	2018-01-06
3	grp2	2018-01-03	2018-01-03
4	grp2	2018-01-07	2018-01-10

This is one approach that makes use of the lag function, following by a cumulative (windowed) sum to create the groups.

with lagged as (
    select
        dt
        , grouping
        , lag(dt) over (partition by grouping order by dt) as lagged_dt
        -- the rn preserves row order for the next step, which is sometimes desired
        , row_number() over (partition by grouping order by dt) as rn
    from seq_demo
), grouped as (
    select
        dt
        , grouping
        , lagged_dt
        , (dt - lagged_dt) as days_diff
        , case when lagged_dt is null or (dt - lagged_dt) > 1 then 1 else 0 end as changed_dt
        , sum(case when lagged_dt is null or (dt - lagged_dt) > 1 then 1 else 0 end)
           over (partition by grouping order by rn rows between unbounded preceding and current row) as dt_group
        -- note on postgresql "rows between unbounded preceding and current row" was not required

    from lagged
)
select
    grouping
    , dt_group
    , min(dt) as start_dt
    , max(dt) as end_dt
from grouped

group by
    grouping
    , dt_group

order by
    grouping
    , dt_group
;

My results:

The specific use case this solved was to identify isolated transactions that had a group count of 1.

This pattern also works well to condense repeated dimensional data that appears in a transactional form.

Hello World

Wed, 03 Oct 2018 14:29:11 +0930