This is a problem a customer presented us. I do not know other scenarios under which this table is used, so we cannot comment on what the primary key should be.

Regardless, I agree with your point that tables should have primary keys defined

thank you so much for the update. I’m glad to see the query suggestions did a lot to improve performance.

I am still wondering about the primary key for the atable table. In particular, I am wondering whether my assumptions that is was {id, off, len, name} is correct. (I mean conceptual pk – I didn’t realize at first but you guys were correct to point out that name can’t fit into a MyISAM index)

A richer syntax, in which some fields would be part of the key definition in a secondary index (in this case only (id, off, len) are needed to define the sort order) and other fields (name, in this case) are used to cover the query, would be cleaner, and you’d end up with a smaller key.  

There are other databases that allow for such secondary index definitions.  I discussed this somewhat in a blog posting on clustering indexes.  

So I think that in the MySQL case, long keys are the price one pays for covering indexes.

O is for the original posting.
BR is for Bjorn/Roland.
S is for Steven.

Original Bjorn/Roland Steven
MyISAM non-covering
O: 17.149s BR: 3.38s S: 9.02s

TokuDB non-covering
BR: 3.63s S: 9.16s

TokuDB covering
O: 4.9s BR: 1.36s S: 7.02s

Not the easiest to read, and if I figure out to post tables I will.

Original Bjorn/Roland Steven
MyISAM non-covering 17.149s 3.38s 9.02s
TokuDB non-covering 3.63s 9.16s
TokuDB covering 4.9s 1.36s 7.02s

A couple of notes: (1) Bjorn’s and Roland’s suggestions are almost identical, and the gave the same times. (2) Steven reports that he gets the same running time with MyISAM both with and without the non-covering index defined about. My results are very different: MyISAM with no index takes 84s. I don’t know how to explain the discrepancy.

Conclusion: (1) Query optimization makes a big difference in the run time. (2) Covering indexes also can make a big difference.

Many thanks to everyone else for the great comments! We’re running tests on the actual data with the various suggestions and we’ll post the results soon.

The design problems start in the base table, but sometimes those issues are harder to fix in the short term. Here a quick fix is to think about what the query actually has to do in order to produce the required result set. Most importantly we don’t need to join each and every combination – only the unique combinations. The following query works well with or without an additional index. Slightly better with.

SELECT a.name,
Count(pd1.name) AS CountOfe2
FROM
atable a
inner join (select
name,id,offset,len
from
atable
group by
name,id,offset,len
) pd1
ON (a.id = pd1.id)
AND (a.offset = pd1.offset)
AND (a.len = pd1.len)
WHERE ((a.name<>pd1.name))
group by a.name

NOTES:
1) I had to fabricate a dataset, different datasets produce different times. I had to come up with a moderately poor dataset since several sets caused MySql to take in excess of 5 minutes.
2) The original query took 174 seconds on my windows laptop (dual core 2ghz w/2gb memory) running mysql 5.1.
3) The modified query took .984 seconds without a composite index and .928 seconds with it.
4) SQL Server managed 34 seconds for the original query without a composite index and less than a second for all other combinations.
5) Both the original query and the modified query return the same resultset.

The original SQL with its nested scalar queries looks to have written by someone with a background in a database which has a optimizer which can handle this better.

When I see this type of join the table on itself stuff it is screaming poor design. You shouldn’t need to de-dupe atable.name in this manner.

Can I guess, this is reporting unique names and counts from a coordinate/tree/matrix table?

Have Fun

SELECT
atable.name,
COUNT(DISTINCT pd1.name) CountOfe2
FROM
atable
INNER JOIN
atable pd1 USING (id, off, len)
WHERE
atable.name <> pd1.name
GROUP BY
atable.name
ORDER BY
CountOfe2 DESC;

I wonder, how does that compare to your solution for the customer’s dataset?

With an index on (id, len, off) and some (probably bad) artificial data (22k rows), I get:

The original query: 17404 rows in set (34.01 sec)
The simple version: 17404 rows in set (1.40 sec)

So it’s faster by a factor of 24.3 for me.

Using a bigger dataset (32k rows) I get:
The original query: 27460 rows in set (1 min 23.77 sec)
The simple query: 27460 rows in set (2.62 sec)

So we got a factor of about 32 now. That the gains increase with a growing dataset is probably also due to the original query getting pretty huge temporary tables (the test with 32k rows used almost 3GB worth of temp. tables here), while the simple query is pretty modest using well below 30MB for its temporary table (And I’m on a slow RAID-1). At least for the random dataset I experimented with.

Interestingly, with my dataset, the original query doesn’t see any performance improvements at all when adding the the index on (id, off, len) compared to the table created by the DDL statement you gave. So I guess my testing dataset is pretty different from the actual data you’re working with (who would have guessed ).

Any chance to get some real number for the simple query? Would be interesting to see how that compares to your solution.

Of course, this is all assuming that the query you’ve shown is close enough to the real thing to make the simple version applicable.