Thursday, March 31, 2016

MySQL, auto_increment and the magic number 0

Ran into a MySQL issue at work recently. My group created three static lookup tables which we populated via handcoded insert statements. For this kind of lookup table, we like having control over the primary key (`id`), so as a general rule we explicitly set this column, rather than relying on auto_increment. For example a table might look like the following (this is a made-up example, as I try to maintain separation of work and blog):

EMPLOYEE_TYPE (id, type)
0 Unknown
1 Manager
2 Grunt
3 Overlord

We create the table then insert like so:

create table EMPLOYEE_TYPE (id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, type VARCHAR(20));

insert into EMPLOYEE_TYPE (id, type) values (0, 'Unknown');
insert into EMPLOYEE_TYPE (id, type) values (1, 'Manager');
insert into EMPLOYEE_TYPE (id, type) values (2, 'Grunt');
insert into EMPLOYEE_TYPE (id, type) values (3, 'Overlord');

(We thought it would be a nice convention to use 0 to represent 'Unknown' across several tables.)

When we kicked off this script, it never made it beyond the second statement because:

mysql> insert into EMPLOYEE_TYPE (id, type) values (1, 'Manager');
ERROR 1062 (23000): Duplicate entry '1' for key 'PRIMARY'

MySQL first overrode our 0 value for 'Unknown', setting it to 1. Then we tried to insert 'Manager' with id 1 and BOOM! Primary Key violation. It turns out MySQL has an oddball special case for auto_increment columns. If you explicitly specify your id, it is used instead of the auto_increment. Fine. Perfect. UNLESS THAT NUMBER IS 0, IN WHICH CASE, 0 IS IGNORED AND THE AUTO_INCREMENT VALUE IS INSTEAD USED. WHAT THE--??

What is the meaning of this? From a page in the documentation:
No value was specified for the AUTO_INCREMENT column, so MySQL assigned sequence numbers automatically. You can also explicitly assign 0 to the column to generate sequence numbers. If the column is declared NOT NULL, it is also possible to assign NULL to the column to generate sequence numbers. When you insert any other value into an AUTO_INCREMENT column, the column is set to that value and the sequence is reset so that the next automatically generated value follows sequentially from the largest column value.
Ok so at least it's documented, but wow, there are such odd design decisions in MySQL sometimes! Fortunately they often provide configurable database parameters to disable them. And this strange case is no exception in that one can set sql_mode to NO_AUTO_VALUE_ON_ZERO to disable this loony behavior. Unfortunately I find nobody ever bothers to change these kinds of parameters from the defaults, and the defaults tend to err on the crazy side.

Saturday, March 26, 2016

JUST. BIND.

I saw this BBC article yesterday - These unlucky people have names that break computers - it's about people with names such as Jennifer Null who have problems inputting their name into various systems because, you know, null.
But to any programmer, it’s painfully easy to see why “Null” could cause problems for a database. This is because the word “null” is often inserted into database fields to indicate that there is no data there. Now and again, system administrators have to try and fix the problem for people who are actually named “Null” – but the issue is rare and sometimes surprisingly difficult to solve.
Poor Jennifer Null - always at the mercy of poor programming. But the ghost of my favorite Tom Kyte post always returns when I hear of this kind of problem. JUST BIND!!

I don't know if it's difficult to deal with in Javascript or other parts of a modern system, but the database tier should have no problem. It doesn't matter if you have "NULL" in your name, it doesn't matter if your name is Bobby Tables or has any number of quotes, semicolons or reserved words. It doesn't matter if it contains the string DROP DATABASE or DELETE if...YOU. JUST. BIND.





Saturday, March 05, 2016

Dan Martensen's blog: SQL Performance of Join and Where Exists

Here's a neat little article on the blog of a fellow named Dan Martensen. I really enjoy performance experiments that show how one approach fares vs. another - it's something I do myself as well. :) This one shows how "where exists" performs vs. an inner join in Postgres 9.5.

Here is the article.

It turns out "where exists" is faster, because as the explain plan reveals, a join performs a Merge Join step, while "where exists" can get away with a Merge Semi Join step. What do these steps mean? Let's turn to the documentation (Source):
  • merge join: Each relation is sorted on the join attributes before the join starts. Then the two relations are scanned in parallel, and matching rows are combined to form join rows. This kind of join is more attractive because each relation has to be scanned only once. The required sorting might be achieved either by an explicit sort step, or by scanning the relation in the proper order using an index on the join key. 
As for Merge Semi Join, I can't find such a clear definition in the docs, but Dan Martensen's article (that I linked to above) mentions: "It’s worth noting Where Exists doesn’t need to return a value to the parent query in the SELECT. It’s WHERE clause compares parent to subquery ids using a boolean short circuit early exit."

So maybe that's a clue as for what goes on under the hood -- it just checks for the matching rows' existence then quits. No data retrieval required.

Perhaps I will soon check if MySQL behaves similarly, as my work will involve more MySQL soon!

Saturday, February 27, 2016

Your comments and questions wanted!

So once in a while I'll take a look at my blog statistics, and while I don't exactly have a vast readership, I do find that my most-read post by far is Postgres Query Using Window Functions. Here are the stats:


 

I'd like to write more on this apparently popular topic - I just need some inspiration for specific questions and/or use cases to experiment with. So if anyone reading this has a specific question on window functions (or anything else for that matter), I'd be happy to attempt to answer!

Feel free to just ask away in the comments!

Wednesday, February 24, 2016

From the "Databases at CERN" blog...

"My experience testing the Oracle In-Memory Column Store" The post is from 2014 but I just stumbled across it Googling for benchmarks for the Oracle In-Memory Column Store...Just purely out of curiosity. Cool to see that CERN made good use of it! Though I can't say I understand the physics analysis query they use as an example. :D

Friday, February 05, 2016

mwrynn PL/SQL table API now on github!

See title. :)

https://github.com/mwrynn/plsql-table

This inspires me to work on it some more - clean it up, document better, add new stuff. (Would really love to add column groupings/tags. i.e. tag a bunch of columns as "dimension" for example, and tell call an easy-peasy function to just "grab all columns tagged as 'dimension'"...

Thursday, January 07, 2016

Postgresql 9.5...released!

Check out some of these fabulous features of Postgres 9.5 if you haven't already: http://www.postgresql.org/about/news/1636/

We finally have upsert. Finally have CUBE, ROLLUP and GROUPING SETS (I really wanted these when I was playing around with Retrosheet baseball data). We also have BRIN indexes for indexing large-scale tables. And there's now even a way to interface with...dun dun dun...BIG DATA DATABASES.

Thursday, December 17, 2015

Reddit post: Postgres Table inheritance and reporting queries

Hello! It's been a while. I just thought I'd share a post I wrote on reddit yesterday. A poster asked for help regarding a Postgres conundrum, and I thought that Postgres's table inheritance feature would fit the bill quite nicely. He was happy with the suggestion and ran with it. Woo!

Link to Reddit thread - my reply is below the original post. (My username is mwdb)

Friday, May 29, 2015

Thursday, March 05, 2015

The mwrynn PL/SQL Table API

The mwrynn Table API is a PL/SQL object type I’ve written for Oracle. It wraps up some handy functions for generating SQL queries, and performing common operations such as “disable all indexes” that Oracle doesn’t provide conveniently out of the box. (I made it an object type instead of a package so it can be extended for a particular table, to the user's liking.) 

My initial inspiration was the the many times I’ve wanted the following construct to be available in SQL: Imagine you had a 50 column table, and you wanted to select col1 through 48, but not 49 and 50. I want to do this: SELECT *-(col49, col50) FROM mytable.

Typing it all out is Tedium City: SELECT col1, col2, col3, col4, col5, col6, col7, col8, col9, col10, col11, col12 [snip — we’re just 25% there]

At the same time, “SELECT *” can be a bad practice to use in your code. Generally fine for running ad-hoc queries, but I wouldn’t want to do that in a PL/SQL package that I plan on using in prod. So, my Table API would provide the entire select list (col1, col2, etc. etc.) for you, which you could just copy and paste. 

It kind of snowballed from there. I’ve got a number of functions, for example I built one that matches column names between two tables. This could be useful for merge/update statements — your external table linked to a flat file probably has column names matching the destination you want to copy to. So, this function would generate a.col1=b.col1, a.col2=b.col2, etc.

Another neat function generates random data. It automatically reads each column’s data type, and generates the appropriate type of data. I’ve found this useful for testing purposes, particularly although it has its limits of course (completely random values tend to have a high cardinality for example — maybe I’ll add optional cardinality constraints in the future!) I’ve found in the case of dates and timestamps it’s useful to have a date range limit, so I've implemented that. For numbers and strings well, there are no constraints at the moment. Foreign keys are generated by selecting a random PK value from the parent table.

Enough talk. Let’s see some action. Let’s run this PL/SQL block. Comments explain what I’m doing at each step. But first, we need a little DDL

 
 CREATE TABLE state (id VARCHAR2(2) PRIMARY KEY, name VARCHAR2(30));  
 INSERT INTO state SELECT 'NY', 'New York' FROM dual;  
 INSERT INTO state SELECT 'CA', 'California' FROM dual;  
 INSERT INTO state SELECT 'FL', 'Florida' FROM dual;  
 INSERT INTO state SELECT 'CO', 'Colorado' FROM dual;  

 CREATE TABLE person (id INT PRIMARY KEY, first_name VARCHAR2(10), last_name VARCHAR2(10), address VARCHAR2(20), city VARCHAR2(10), state VARCHAR2(2) REFERENCES state(id), zip VARCHAR2(5))   
 CREATE INDEX person_city_idx ON person(city);  
 CREATE INDEX person_zip_idx ON person(zip);   

 CREATE TABLE person_ext (id INT PRIMARY KEY, first_name VARCHAR2(10), last_name VARCHAR2(10), address VARCHAR2(20), city VARCHAR2(10), state VARCHAR2(2), zip VARCHAR2(5))   
 DECLARE  
  person_table_obj table_obj;  
  person_ext_table_obj table_obj;  
  cnt INTEGER;  
BEGIN
  --create table objects for mwrynn.person mwrynn.person_ext; third param null means local table, i.e. no dblink is used in qualifying table name
  person_table_obj := table_obj('PERSON', 'MWRYNN', null);
  person_ext_table_obj := table_obj('PERSON_EXT', 'MWRYNN', null);

  --display all the columns
  dbms_output.put_line(person_table_obj.all_cols);

  --disable all indexes and display number of successful disables
  dbms_output.put_line('Disabled ' || person_table_obj.disable_indexes || ' indexes.');

  --re-enable the indexes, and display number of successful renables
  dbms_output.put_line('Enabled ' || person_table_obj.enable_indexes || ' indexes.');
  
  --get list of columns to join on
  dbms_output.put_line(person_table_obj.join_list(person_ext_table_obj, 'a', 'b'));
  
  --get the list of matching columns, i.e. a.col1=b.col1, a.col2=b.col2, etc.
  dbms_output.put_line(person_table_obj.matched_update_list(person_ext_table_obj, 'a', 'b'));
  
  --generate random data, just 5 rows for our purposes. The foreign key column, state, will be looked up in the parent lookup table 
  cnt := person_table_obj.insert_random_rows(5, null, null); 
END;
 /  
 SELECT * FROM person;  

The output:

 anonymous block completed  
 ID,FIRST_NAME,LAST_NAME,ADDRESS,CITY,STATE,ZIP  
 Disabled 2 indexes.  
 Enabled 2 indexes.  
 a.ID=b.ID  
 a.FIRST_NAME=b.FIRST_NAME,a.LAST_NAME=b.LAST_NAME,a.ADDRESS=b.ADDRESS,a.CITY=b.CITY,a.STATE=b.STATE,a.ZIP=b.ZIP  
     ID FIRST_NAME LAST_NAME ADDRESS       CITY    STATE ZIP   
 ---------- ---------- ---------- -------------------- ---------- ----- -----  
 5041759599 DrUeedsMza cmUybFRdok ZyLBSrpNarfAsnNbHxzK xTkXtWdngA CA  bWjJs   
  885481864 VXNsOpNbwb POwZITUopy sqFQHHWcxFfcVNCLJyzq ysCdZKLKEQ FL  UcfkU   
 2357440692 vtDRoSaFXI JThrcRBQbL YrEyCAEAxXYErDInLciw SVdmHxtCNw CO  KxuIS   
 7331548629 pyKOLSGsTL NTlFxgpxAQ wsEZwiBlLaZAbsLenkeb IHWzBljRVL FL  LlhLA   
 4568868701 sftvDZyZpc HxEWPyenqR uPcMxkmJEMCgPYXMgUYq OrOZidpfKB CA  kpPgE   

First we've disabled and re-enabled the two indexes. (The primary key's index is never disabled.) Then we can see the join condition. Next is the output of matched_update_list -- the list of matching columns for the purpose of UPDATEs and MERGEs.

Finally we see the randomly generated data. As you can see, the values for the state column were pulled from the parent state table. The rest of the data is hideous but that's all it is -- random nonsense. If I had a lookup table of city/state/zip combinations, it could draw from that one. But I don't. :)

That's the mwrynn Table API for now. With such few columns this might not look so amazing, but it can really save time. I once had to work with about a dozen tables, each with about 20-40 columns, and write out my insert queries and such, plus generate test data. There are GUI tools that may provide some of the generation functionality, but I always found them cumbersome to use. All in all, this object type saved me quite a number of hours!

Note: I've added this API to Github! https://github.com/mwrynn/plsql-table