Browsing All Posts published on »October, 2017«

Custom Window Function in Spark to create Session IDs

October 29, 2017 by


(note: crossposted from my┬áNuvolatech Blog If you’ve worked with Spark, you have probably written some custom UDF or UDAFs. UDFs are ‘User Defined Functions’, so you can introduce complex logic in your queries/jobs, for instance, to calculate a digest for a string, or if you want to use a java/scala library in your queries.